Unmanned aerial vehicle rooftop inspection system

ABSTRACT

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for an unmanned aerial system inspection system. One of the methods is performed by a UAV and includes receiving, by the UAV, flight information describing a job to perform an inspection of a rooftop. A particular altitude is ascended to, and an inspection of the rooftop is performed including obtaining sensor information describing the rooftop. Location information identifying a damaged area of the rooftop is received. The damaged area of the rooftop is traveled to. An inspection of the damaged area of the rooftop is performed including obtaining detailed sensor information describing the damaged area. A safe landing location is traveled to.

CROSS-REFERENCE TO RELATED APPLICATIONS

Any and all applications for which a foreign or domestic priority claimis identified in the Application Data Sheet as filed with the presentapplication are hereby incorporated by reference in their entirety under37 CFR 1.57.

BACKGROUND

Inspecting properties (e.g., apartment buildings, office buildings,single family homes) for damage (e.g., weather damage), or otherreasons, can include significant time investments by personnel trainedto perform the inspection. Additionally, typical inspection proceduresinvolve physically climbing onto roofs and ledges, incurring significantrisks of both inspection personnel injury and damage to the propertyitself. Subsequently, companies involved in inspection generally need todevote substantial time training personnel, and subsequently ensuringthat the personnel follow proper safety and governmental procedures.

SUMMARY

Particular embodiments of the subject matter described in thisspecification can be implemented so as to realize one or more of thefollowing advantages. Utilizing an unmanned aerial vehicle (UAV), anentity (e.g., a company, a governmental entity) can schedule inspectionjobs, and provide the jobs intelligently to one or more UAVs to performinspections of potentially damaged properties (e.g., a home, anapartment, an office building, a retail establishment, etc.). Byintelligently scheduling jobs, a large area can be inspected usingUAV(s), which reduces the overall time of inspection, and enablesproperty to be maintained in safer conditions. Furthermore, by enablingan operator to intelligently define a safe flight plan of a UAV, andenable the UAV to follow the flight plan and intelligently react tocontingencies, the risk of harm to the UAV or damage to surroundingpeople and property can be greatly reduced.

In general, one innovative aspect of the subject matter described inthis specification can be embodied in methods that include the actionsof receiving, by the UAV, flight information describing a job to performan inspection of a rooftop; ascending to at least a particular altitude,and performing an inspection of the rooftop including obtaining sensorinformation describing the rooftop; subsequent to inspecting therooftop, receiving location information identifying a damaged area ofthe rooftop; traveling to the damaged area of the rooftop; performing aninspection of the damaged area of the rooftop including obtainingdetailed sensor information describing the damaged area; and travelingto a safe landing location.

The details, including optional details, of one or more embodiments ofthe subject matter of this specification are set forth in theaccompanying drawings and the description below. Other optionalfeatures, aspects, and advantages of the subject matter will becomeapparent from the description, the drawings, and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A illustrates an example of an Unmanned Aerial Vehicle (UAV)performing a first flight to perform rooftop damage inspection of aproperty.

FIG. 1B illustrates an example of a UAV performing a new flight plan toperform rooftop damage inspection of a property.

FIG. 2 illustrates a block diagram of example systems utilized inperforming a rooftop inspection of a property.

FIG. 3 is a flowchart of an example process for determining informationdescribing a job for a UAV to perform an inspection of a rooftop.

FIG. 4 is a flowchart of an example process for determining flightinformation to perform a job.

FIG. 5 is a flowchart of an example process to perform an initialinspection of the rooftop.

FIG. 6 is a flowchart of an example process for performing a subsequentinspection of the rooftop.

FIG. 7 is a flowchart of an example process to determine a visualclassifier to identify rooftop damage.

FIG. 8 illustrates an example process for performing an inspection of arooftop using the visual classifier.

FIG. 9 illustrates a block diagram of an example flight control systemarchitecture for a UAV.

FIG. 10A illustrates an example user interface for assigning waypoints,and designating actions to take at each waypoint.

FIG. 10B illustrates an example user interface showing a representationof the UAV 10 implementing a flight plan.

Like reference numbers and designations in the various drawings indicatelike elements.

DETAILED DESCRIPTION

This specification describes systems and methods to perform inspectionsof properties for damage (e.g., weather damage, structural damage, andso on) using an Unmanned Aerial Vehicle (UAV). While the specificationdescribes determining damage of rooftops as an example, the system andmethods can be utilized to determine damage to any portion of a property(e.g., home, building, apartment, multi-unit home, factory, bridge,power plant, and so on). In this specification, UAVs include anyunmanned aerial vehicles, such as drones, unpiloted aerial vehicles,remotely piloted aircraft, unmanned aircraft systems, any aircraftcovered under Circular 328 AN/190 classified by the International CivilAviation Organization, and so on. For example, the UAV may be in theform of a single or multi-rotor copter (e.g., a quad-copter) or a fixedwing aircraft. In addition, certain aspects of the disclosure can beutilized with other types of unmanned vehicles (e.g., wheeled, tracked,and/or water vehicles).

A described cloud system can generate information describing one or morejobs to be performed by a UAV, optionally in concert with an operatorusing a user device (such as a ground control system, including alaptop, tablet, and so on) that is in communication with the UAV. Inthis specification, a job is any task to perform an inspection ofproperty for damage, and the cloud system can intelligently batch jobstogether such that a same UAV can inspect a grouping of properties in asame geographical area (e.g., a city block, a suburban area), therebyreducing the number of separate flights needed, and reducing energyconsumption that would otherwise be needed to repeatedly place the UAVin the geographical area. A job, or job information, can be provided toa UAV, or user device, with sufficient information to enable the UAV,and/or user device, to implement the job, and can also be known as aflight plan. The cloud system can receive user input on one or more userinterfaces generated by the cloud system, such as in an interactivedocument (e.g., a web page) provided for presentation on a user device.A user of the cloud system can provide information describing one ormore properties to be inspected (e.g., an address of each property), andthe cloud system can determine information associated with one or morejobs to inspect the properties.

To determine information associated with a job, the cloud system canaccess property information for an input property (e.g., from publicand/or private databases via an application program interface (API)),and determine a property boundary for the property. The propertyinformation received by the cloud system can be of various data types,for example, parcel polygons, vector, rasterized, shape files, or otherdata types. For the particular property, the cloud system may create thegeofence envelope based on the property shape data. The various datatypes ideally would have geolocation and/or coordinate information, suchas latitudinal/longitudinal points for use in orienting and creating thegeofence envelope. The geofence envelope may be identical in shape tothe property boundary. Optionally, the boundary of the property may bereduced in size. For example, the property boundary may be reduced insize by a set distance, for example 5 meters, towards a centroid of theproperty. Reduction of the geofence envelope creates a buffer zone. Thebuffer zone may help avoid an unintentional flyover of an adjacentproperty boundary. Optionally, the cloud system may display an area withparcel polygonal data. An interface of the cloud system may then receivea selection of one or more parcels. The cloud system then can use theselections to create one or more jobs, and multiple geofence envelopes.For the multiple parcels, the UAV would be taken to each parcelproperty, and the operator would conduct each job with the UAV for therespective parcel property. Additionally, the UAV can conduct multiplejobs for separate parcel properties if the parcels are close enough inproximity, and the UAV has sufficient fuel or battery power to conductthe multiple jobs. The cloud system can then determine a geofenceenvelope that is limited or substantially limited by the propertyboundary, ensuring that a UAV is to remain within the property boundary.In this specification, a geofence envelope is a virtual perimeter, orvolume of space, for a real-world geographic area, or volume. Therefore,a 2D or 3D geofence envelope limits locations of a UAV to the real-worldgeographic area or volume, and a UAV can include one or more systems toreceive a specified geofence, and enforce the geofence.

Similarly, the cloud system can determine, or obtain informationidentifying, a boundary of the rooftop of the property to be inspected.For instance, a user of the cloud system can access imagery (e.g.,satellite imagery, or images obtained from prior jobs) of the propertyto be inspected (e.g., from a public or private data store via an API),and can interact with the imagery to indicate the boundary of therooftop. As an example, the user can select corners of the rooftop todescribe a polygon that encompasses the rooftop. In another instance,the cloud system can determine (e.g., using machine vision techniques tocluster potential rooftops found in the imagery, and using machinelearning algorithms to correctly label the actual rooftop) the boundaryof the rooftop. Furthermore, optionally if the property has previouslybeen inspected, the cloud system can access stored informationdescribing the rooftop boundary determined in the prior inspection. Aswill be described, the boundary of the rooftop may be utilized to informa flight pattern of a UAV.

The cloud system can access from one or more sources of weatherinformation, including weather forecasts for an upcoming period of time,solar light models, and configuration information associated with one ormore UAVs, to determine an optimal UAV and time period for the UAV toperform a job. For instance, the cloud system can determine that due toweather effects (e.g., rain) on an upcoming day, the job is betterscheduled on a day with sun. Additionally, the cloud system candetermine that sensors included a UAV will have better performance withparticular weather (e.g., electro-optic imaging sensors can performbetter at noon on a sunny day, as image contrast will be maximized, andshadows will be minimized).

For autonomous flight of the UAV, a flight plan may be created andtransmitted to the UAV from the cloud system or a user device (describedbelow). The flight plan instructs the UAV with regard to autonomousflight along a particular flight path in three-dimensional space. Theflight plan includes a series of connected waypoints that define wherethe UAV should fly and what actions the UAV should take during aparticular flight. The UAV may have an autopilot flight module operatingon a UAV computer system that uses the flight plan to automaticallypilot the UAV.

Additionally, flight plan contingencies may be created using the cloudsystem or user device. A flight plan contingency instructs the UAV toperform an action or operation based on certain contingency criteria.For example, contingency criteria may be the detection of a low batteryor fuel state, or malfunctioning of an onboard sensor, motor, or adeviation from the flight plan, or crossing over a geofence boundary bythe UAV. Other contingency events may include ground control systempower or system failure, lost or degraded telemetry link to/from the UAVand ground control system, stuck motor, GPS failure or degradation,autopilot sensor failure (e.g. airspeed, barometer, magnetometer, IMU),control surface failure, gear failure, parachute deployment failure,adverse weather conditions, nearby aircraft in airspace, vehiclevibration, aircraft fly-away).

A UAV computer system, or optionally a user device of an operator, canreceive information describing the job, and perform the associatedinspection (e.g., the user device can provide information describing thejob to the UAV, or the UAV can receive information from the cloud systemand be brought by the operator to the property). The user device mayreceive flight plans from the cloud system for transmission to the UAV.The user device also allows for manual override of a UAV operating in anauto-pilot mode. The user device may transmit a flight plan to the UAVeither via a wireless or tethered connection. Ideally, the user deviceis a mobile device, such a laptop, mobile phone, tablet device, withradios for data communication. After, or while, performing theinspection, the UAV can provide images, and other sensor information, tothe cloud system or user device. Ultimately, an interactive report canbe generated (e.g., by the cloud system, user device, or UAV) thatincludes summary data associated with the job, which can optionallyinclude a graphical representation of the property and/or rooftop withdamaged areas identified (e.g., highlighted), the types of damagedetected, and which provides access to the raw image and sensorinformation (e.g., a user can select a damaged area and accessassociated image and sensor information). Sensors can include visibleand/or non-visible light cameras, RF sensors, chemical sensors, soundsensors, spectrometers, magnetometers, radiometers, wind sensors,ambient light sensors, barometers, temperature sensors, thermal imagers,range sensors, and/or other sensors, and so on. The UAV can store thereal-world information along with metadata (e.g., time informationassociated with each activation, position and attitude data of the UAV,distance of the UAV from the structure, wind information, ambient lightinformation, and so on). In this specification, real-world information,or sensor information, includes any measurement, reading, image, audio,and so on obtained from a sensor that describes, characterizes,measures, and so on, a real-world area.

As described below FIGS. 1A-1B, optionally the inspection can beperformed in multiple steps, an initial operation during which a UAV cantravel over the property and obtain sensor information (e.g., images),and a subsequent operation during which the UAV can receive (e.g., froman operator using a user device such as a ground control system)indications of specific locations of damage or likely damage on therooftop, and can descend over each location to obtain detailed sensorinformation of the location. Interacting with a user interface of theuser device, an operator can identify locations of damage or likelydamage after the user device receives the sensor information from theinitial flight operation, and the identified locations can be providedto the UAV to effect the subsequent operation.

Optionally, the initial flight operation can be performed with a firstsensor included with the UAV (e.g. a camera with a wide-angle lens, suchas an effective full-frame focal length of 16 mm, 20 mm, 24 mm, 35 mm).The subsequent flight operation can be performed with a second sensorincluded in the UAV (e.g., a camera with a longer focal length, such as50 mm, 85 mm, 125 mm). As will be described, the initial flightoperation can benefit from a wide-angle lens (e.g., the UAV can capturea wider area, and also obtain more of the sides of properties which canbe used when generating 3D models of the property). The subsequentflight operation can benefit from a longer focal length (e.g., to obtainclose detail of a damaged area without having to descend close to thedamaged area).

FIG. 1A illustrates an example of an Unmanned Aerial Vehicle (UAV) 10performing a first automated flight plan to perform rooftop damageinspection of a property 20. The UAV 10 computer system has receivedinformation describing the property 20, including a property boundary14, and a boundary of the rooftop 30 to be inspected (e.g., corners30A-30D of a polygon encompassing the rooftop).

The UAV 10 may be configured to travel (e.g., flying) at a safe distanceor height 16 (e.g., above ground level altitude, or mean sea levelaltitude) over the rooftop 30, with the safe distance indicated by theoperator 12 using the ground control system as being a distance at whichno obstructions can interfere with the UAV 10 as it travels within thegeofence property boundary 14. Optionally, the ground control system ora cloud-based system can determine the safe distance (e.g., usingimagery or 3D models of the property 20). For example, the 3D models maybe 3D polygonal data models with dimensions of the property 20. Theground control system or UAV determine based on the height of theproperty, and any structures of the property, a safe distance to flyover the property. The distance may be a predetermined distance such as3 meters, or a variable distance. The UAV 10 travels over the rooftop 30at the safe distance 16 according to a flight pattern, which can be azig-zag pattern (e.g., the UAV can travel from corner 30A to corner 30B,turn towards corner 30D, travel back down towards corner 30B shiftedover by a particular distance, and repeat), or the flight pattern can bethat of a different flight pattern (e.g., a flight pattern in which therooftop 30 is traversed in slightly overlapping concentric circles, assimilar to an ice resurfacer). The specific flight pattern of the UAV 10can depend on configuration information of the UAV, for instance afixed-wing UAV can require a greater distance to turn and thus thedifferent pattern can be preferred. The UAV receives the flight patternfrom the ground control system of the cloud-based system. The UAV usesthe flight pattern to fly the pattern in an autopilot mode.

The flight pattern of the UAV 10 (e.g., a number of times the UAV 10 isrequired to turn) can depend on a size of the rooftop 30 (e.g., a largerrooftop can require additional turns) and the UAV configurationinformation (e.g., a camera included in the UAV 10 with a long focallength can require more turns as the camera has less field of view thana longer focal length camera). As will be described, the cloud system,the user device, or the UAV can determine the flight path for theinspection.

In the example of FIG. 1A, the UAV 10 is shown conducting an automatedflight plan over the rooftop 30 while activating sensors included withthe UAV 10. The UAV 10 can activate the sensors periodically (e.g.,after traveling a threshold distance, such as ½ meter, 1 meter, 2meters, or after a threshold time period, such as ½ second, 1 second),or can capture sensor information continuously (e.g., a camera canrecord video, such as 60 frames per second video). Upon the UAV's 10completion of traversing the rooftop 30, the UAV can provide theobtained sensor information (e.g., images) to the operator for review(e.g., on a laptop, tablet, wearable computer, and so on) through awired, or wireless connection (e.g., BLUETOOTH, 4G, LTE, Wi-Fi, and soon). Optionally, the UAV 10 can provide the sensor information to theoperator 12 via the user device as it obtains the sensor information(e.g., the UAV can provide the sensor information in substantiallyreal-time). The sensor information also may be stored onto a removablestorage card used by the sensor, or as configured, from the UAV'sonboard processing system.

As the UAV 10 conducts the flight plan, the user device of the operator12 can present a graphical representation of the progress of the UAV 10.For instance, the user device can present imagery (e.g., satelliteimagery) of the property 20 along with a representation of the flightpattern the UAV 10 is to follow. As an example, the representation ofthe flight pattern can include waypoints the UAV 10 is to navigate to,with each waypoint connected by a line, arrow, or other connectingrepresentation. The UAV 10 can provide geospatial location informationto the user device, and the user device can update a representation ofthe UAV 10 as it travels. Optionally, as the UAV 10 travels and capturessensor information (e.g., digital images), the user device can include abounding quadrilateral, or other shape, around a representation of theUAV 10, which represents a field of view being captured by the UAV 10 insensor information (e.g., digital images), with for example, the fieldof view being an area of the property 20 included in the sensorinformation (e.g., based off capabilities of the camera, such as focallength, sensor resolution and size, lens aperture, and so on). Forinstance, as illustrated in FIG. 10B, a representation 1056 of the UAV10 is traveling to a particular waypoint (e.g., waypoint A), andincludes a quadrilateral 1058 surrounding the representation 1056. Thequadrilateral 1058 tracks the movement of the UAV 10, and can bemodified as the UAV 10 rotates (e.g., rotation in roll, yaw, pitchaxes), shifts forward as the UAV 10 shifts forward, and so on. Thequadrilateral 1058 can be determined from a determined field of view ofa sensor (e.g., camera) with respect to the UAV 10s position (e.g.,height, latitude and longitude) and attitude information. The field ofview can be toggled on and off by the operator 12, and optionally eachquadrilateral captured by the UAV's 10 sensor can be highlighted,colored, shaded, and so on, to indicate that the portion of the property20 has been captured in sensor information (e.g., digital imagery). Inthis way, the operator 12 can monitor the progress of the UAV 10, andensure that the entirety of the rooftop 30 is captured in imagery.

The UAV's onboard processing system may detect sudden shifts or movementof the UAV, for example due to a sudden gust of wind. The onboardprocessing system may detect output of UAV accelerometers and/or UAVrotational rate gyros. If a threshold magnitude level is detected whenan image is taken by the UAV, then the UAV may be configured orprogrammed to retake the image. The UAV processing system may delete theinitial image, or the user device may receive log data from the UAVidentifying the second image as the preferred image to use whenconstructing a mosaic image of the rooftop 30.

A UAV configured for streaming of images to the user device may receivea command from the user device to retake an image. For example, asimages are received by the user device, the user device may display theimage. Quality information about the image may be determined bysharpness measurements of the image. For instance, a frequency domainanalysis of the image can be performed, and a lack of high frequenciescan be indicative of a lack of focus (e.g., compared to an expectedinclusion of high frequencies for the image). Additionally, a laplaciankernel can be convolved with the image (e.g., in the spatial domain) andthe result can be used to determine blurriness of the image (e.g.,intensity values of pixels within a threshold distance can be compared,and a blurry image can have a lack of comparisons greater than athreshold). Additional quality information can include sufficientoverlap with neighboring images, brightness measurements, exposuremeasurements, contrast measurements, and so on. The user device mayautomatically transmit instructions to the UAV to retake an image at agiven waypoint if the image fails to meet an image quality threshold.Optionally, the UAV may have onboard GPU processing capabilities. TheUAV may after taking an image, move into a holding position, and the UAVonboard processing system may analyze the image. If the onboardprocessing system determines that an image does not meet a thresholdquality, the UAV then retakes another picture. The UAV processing systemreruns the image quality review process, and then continues with theflight plan if the image passes the quality threshold.

The operator 12 can review the received sensor information on one ormore user interfaces presented on the operator's 12 user device. Forinstance, the user device can execute an application that is incommunication with the UAV 10, and can present sensor information to theoperator for review. The operator's 12 user device can execute software(e.g., mosaicking software, photogrammetry software) that can combine(e.g., stitch together) the received sensor information, and geo-rectifythe sensor information to obtain, for instance, a continuous image ofthe rooftop 30 mapped to a real-world coordinate frame. Optionally, thereceived sensor information can be used to generate a 3D model of theproperty 20 (e.g., the user device can generate the 3D model, or theuser device can provide the sensor information to the cloud system andreceive information describing a 3D model). Images of the rooftop froman initial flight photo survey can be stitched together, and convertedinto an image map layer showing the rooftop, and then used forsubsequent flight planning of a subsequent photo point survey.

Optionally, for the initial flight photo survey, the flight plan mayinclude a waypoint that is designated about the center of the property,or about the center of a building to be inspected. Ideally, the waypointis set at an altitude high enough, for example 45-60 meters above theground so that the entire building or structure will be in the field ofview of the camera. At this waypoint, the UAV will take one or morephotos of the property, or building. One of these “central” photos maybe used as described below to aid in the identification of potentiallydamaged areas of the rooftop, or be used to assist with visual odometry.That is, the central photo can include the entirety of the property,building (e.g., rooftop) or structure being imaged, or the photos caneach include a portion of the property, building, or structure, beingimaged.

Once the user device imports the data from the first flight, theoperator may validate the quality of that data before the user devicebegins stitching the images together (in the case of multiple imagesbeing taken). A user interface of the user device, may provide viewingof each of the individual images such that the user may visually inspectthe images. Optionally, the operator can interact with the user deviceto indicate that the image is problematic (e.g., the user device canpresent selectable options, the operator can swipe left or right on animage, the operator can tap or double tap an image, the operator canverbally describe the problem, and so on). Also, images may be markedautomatically as potentially problematic, for example the user devicecan present potentially problematic images to the operator for review(e.g., based off one or more image quality scores being less than athreshold). An image quality score may be assigned to each of the imagesbased on evaluation of the image by evaluating the generated stitchedmosaic image for continuity (e.g., image features properly segue betweenimages), sufficient overlap of the image with another image, measures ofimage distortion, measures of geographic location error (e.g., a measuredescribing an accuracy of the location coordinates, such as GPS,associated with the image, which can be determined from a field of viewincluded in the image in comparison to other images and/or estimatedfields of view for the location). Also, a composite image quality scoremay be calculated by determining a measure of central tendency (e.g., amean) of the above image quality scores, a weighted average of the imagequality scores, a determining of the minimum and maximum of all imagequality scores, and so on. The user device may indicate that anotherphoto survey is necessary if the overall composite score reaches athreshold number, or if there are a predetermined number of images thatare potentially problematic (e.g., as identified by the operator or fromthe image quality scores). The user device may then use photos from theprevious flight to generate a quick-stitch of the roof-top.

A flight plan for a subsequent flight for a photo survey may be createdwith the user device. To expedite the creation of the flight plan,information from the first job, or flight plan may be used. For example,for the subsequent flight plan for a photo point survey, the flight planmay reuse the same takeoff position, landing position, geofence,contingencies and initial safe altitude from the first flight plan.

The operator 12 can interact with the user device to indicate locationson the rooftop 30 that are determined, by the operator 12, to bepotentially damaged. The user device displays the central image, orstitched-image, or separate individual images. The images may begeo-rectified so that for indicated locations, a geospatial referencelocation may be identified. For instance, the user device can receive atouch selection, mouse selection, or by a pen-input device, by anoperator 12 (e.g., a location on the rooftop 30 can be selected on atouch screen of the user device), and the user device can storeinformation identifying the selected portion (e.g., the image area, andassociated geospatial position (e.g., geospatial reference), such asglobal positioning system (GPS) longitude/latitude coordinates. As willbe described with respect to FIG. 1B, the UAV 10 can receive a newflight plan with the identified damaged locations as waypoints andobtain detailed sensor information of each location.

A flight plan for a subsequent flight for a photo survey may be createdwith the user device. To expedite the creation of the flight plan,information from the first job, or flight plan may be used. For example,for the subsequent flight plan for a photo point survey, the flight planmay reuse the same takeoff position, landing position, geofence,contingencies and initial safe altitude from the first flight plan.

The operator 12 can interact with the user device to set a groundsampling distance (GSD) for each of the photos for the identifiedpotentially damaged locations individually, as a group, or as a wholefor all of the potentially damaged locations. The GSD indicates anacceptable number of number of image pixels/distance (e.g., meter),ensuring that captured images include sufficient detail. To ensure thatimages are captured with sufficient detail indicated by the GSD, theuser device can determine a particular altitude (e.g., an inspectionaltitude) above an object or surface being imaged that will providecaptured imagery with sufficient detail. The particular altitude cantherefore be based off configuration information of included cameras(e.g., a number of pixels, resolution, and so on, of a sensor includedin a camera of the UAV), and indicates a minimum altitude above anobject, or surface, being imaged to capture images with sufficientdetail to satisfy the GSD. A desired GSD for each of the individualphotos taken during the point inspection can be based on a specificdistance from the roof. The UAV computer system may determine thisdistance from the roof top while in flight via one or more distancemeasuring devices, such as onboard Lidar, Leddar, sonar or othermeasuring device used by the UAV. For the new flight plan, the userdevices uses geospatial waypoints that are based on the selected damagedlocations with an associated altitude, or a distance from the rooftop.

For the new flight plan, the operator 12 may interact with the userdevice to set a safe altitude that the UAV will ascend to after eachpoint inspection photo has been taken of a selected damaged location.When the UAV reaches a waypoint for an identified damaged location, theUAV will descend to a distance related to the GSD, and then take apicture of the identified damaged area. The UAV will then ascend to thesafe altitude and then move to the next waypoint for an identifieddamaged location. An initial safe altitude used for the first flightplan may be automatically set as the safe altitude for the second flightplan. The safe altitude, however, should not exceed the ceiling of thegeofence property boundary 14. Ideally, a buffer zone from the ceiling,for example 10 meters, may be used to avoid unintentional geofencecrossing due to altitude measurement sensor errors.

The operator 12 may interact with the user device to set a waypointtransition speed, or the user device may automatically calculate orconfigure waypoint transition speeds. The waypoint transition speed maybe set such that the autopilot maneuvers the UAV from each identifieddamaged location to the next at a safe, but efficient transition speed.The waypoint transition speed may be set as a fixed value, such as 1.5meters per second, or a variable speed based on the distance of onewaypoint to the next waypoint. For example, the waypoint transitionspeed, may be set to a higher speed for waypoints that are fartherapart, and a lower speed for waypoint that are closer apart. Also, thewaypoint transition speeds can be set as function of the useful batterylife, or flight time of the UAV.

Optionally, for any flight plan (independent of damage locationdetermination) an action for a waypoint may be set for determining theslope of a rooftop 12. In this case, for a slope determination waypoint,the UAV may descend at the waypoint to a height above a rooftop 12. TheUAV then may move in a pattern while staying at a constant height (suchas a circle, cross, square) during which the UAV computer system iscapturing the distance (using Lidar, Leddar, or Sonar measurements) fromthe UAV to the surface of the rooftop 12. The slope of the rooftop 12 atthe particular waypoint then can be determined by the respectivelocations and heights recorded for the positions. Also for a waypointfor a damaged location, the slope for the respective portion of therooftop 12 may be determined. Also, the user device or cloud system mayautomatically include waypoints for determine the slope of one or morerooftop 12 sections of a structure. For instance, utilizing propertyinformation describing the property 20, information describingdirectionality of the property 20 can be determined, and utilized todetermine a slope of the rooftop 12. For example, if it is known that afront of the property (e.g., a front door, a driveway) faces aparticular direction, a rooftop can be assumed to slope downwards in adirection orthogonal to the particular direction (e.g., for a drivewaypointing South towards a street, a rooftop can be assumed to slopeupwards to a center point and then downwards when moving from West toEast). The UAV can therefore travel to a center of the property, andtravel West at a same height and take distance measurements.

To effect the slope determination, the UAV can obtain distancemeasurements at points on the rooftop 12 and correlate the distancemeasurements against a 3D position of the aircraft (e.g., locationcoordinates) to determine the slope of the rooftop. That is, the UAV candetermine a point cloud (e.g., a collection of points representing aphysical surface that are measured in a particular reference frame, suchas WGS84) of distance measurements as it travels over the rooftop 12.Optionally, utilizing information describing the property beinginspected, the UAV can determine, or store information describing therooftop

However, utilizing a single-shot measurement sensor, such as a Lidarsensor measuring distance to the UAV at a single location, the UAV isscanning a line on the surface of the rooftop, with the line indicatingdistance information at singular locations along the line. Since theline is not guaranteed to be in a direction of maximum slope on therooftop 12 (e.g., the UAV can be flying along a ridge of the rooftop 12and therefore not detect slope), alternatively other methods can beutilized to determine slope.

For example, the UAV can utilize a planar scanner (e.g., a scanner thatmeasures distance at greater than a single location), in conjunctionwith information describing the UAVs position and attitude, to generatea fuller 3D point cloud. Similar, the UAV can utilize a full Lidarscanner (e.g., a Veloydne Puck) while performing the rooftop 12inspection. Optionally, the UAV can utilize a series of captured imageryto generate a 3D point cloud (e.g., utilizing photogrammetrytechniques). For instance, captured imagery can be correlated togetherto precisely determine locations at which each image was taken relativeto other images (e.g., matching features, fields of view, and so on,included in images). The images can then be analyzed to determinerelative height information of the rooftop 12.

Via a user device interface, the operator 12 can therefore indicate anassociated waypoint for each location determined to be damaged, or forslope determination, and the user device can generate a new flight planthat includes the waypoints. As described above, a waypoint isassociated with a location (e.g., longitude/latitude coordinates), asafe altitude (e.g., a transit altitude identifying an altitude toenforce when traveling between waypoints), an inspection altitude, aminimum distance to remain above the rooftop 12, and so on. Furthermore,as will be described, the UAV travels to each waypoint and descends,from the inspection altitude, towards the rooftop while capturing one ormore images during the descent. The operator 12 can, for each waypoint,one or more actions or maneuvers to be performed by the UAV. For,example, the waypoint may have an action for a number of images tocapture during the descent and/or a time interval to wait betweencapturing images (e.g., the operator 12 can indicate a periodic in time,or distance, value). Additionally, the operator 12 can indicate an orderfor each waypoint, a particular flight pattern (e.g., the operator 12can trace a flight pattern for the UAV to follow), and so on.Furthermore, the operator 12 can indicate a rate for the UAV to descendat a waypoint, for instance a nominal rate, a maximum rate, and so on.Optionally, the user device can determine (e.g., based off capabilitiesof the UAV) a descent rate such that the UAV can descend while remainingsubstantially stable (e.g., ensuring that the UAV is unstable or shakingwhile capturing images).

Optionally, the operator 12 can indicate information associated with aparticular waypoint, and select another location on the rooftop 30 asbeing damaged, and indicate that an associated waypoint should utilizethe same information. In this way, the operator 12 can quickly tap areasof the user device, and the user device can generate all waypointinformation. Optionally, the operator 12 can indicate informationassociated with waypoints (e.g., prior to indicating locations on therooftop 30 as being damaged), and can quickly tap waypoints insuccession, and the user device can utilize the prior indicatedinformation for each waypoint.

Optionally, the operator 12 can indicate that after the UAV travels to aparticular waypoint, or all waypoints), at the safe altitude, the UAV isto hold until receipt of confirmation from the operator 12 to descend.That is, the operator 12 can indicate that the UAV is not free to begin(e.g., without further user input) descending towards the rooftop 12 tocapture imagery. This can be useful in situations where the UAV isrequired to maintain Line-of-Sight with the operator 12. If the operator12 determines that the UAV will be out of Line-of-Sight, and that theoperator 12 has to move locations, the operator 12 can indicate to theUAV to begin descent upon moving to a different location (e.g., the UAVcan receive wireless information from the user device of the operator12). Optionally, in some implementations, the operator 12 can directlycontrol the descent (e.g., control a speed of descent, providecorrections to the descent flight path, or manually control the UAV). Toindicate that the UAV is to hold at the safe altitude prior todescending, the operator 12 can interact with the user device to selectan selectable option associated with a waypoint (e.g., the selectableoptions can include an option for an autonomous photo survey, or for asemi-autonomous photo survey).

For instance, FIG. 10A illustrates an example user interface 1000 forassigning waypoints, and designating actions to take at each waypoint.As described above, an operator 12 can indicate that after traveling toa particular waypoint, the UAV is hold at the waypoint until theoperator's 12 confirmation to descend. As described above, and as willbe described below with reference to FIG. 6, the operator 12 can specifywaypoints associated with damage (e.g., possible damage), and the UAV 12can descend towards the specified waypoint (e.g., descend towards therooftop 30).

To effect the assignment of waypoints for damage inspection, theoperator can interact with the user interface 1000 (e.g., on a userdevice) to specify waypoints in a planning 1002 process (e.g., theoperator 12 has selected an interactable button for making a plan 1002),for instance by interacting with a touch-sensitive screen displaying theuser interface 1000, or by selecting a portion of the user interface1000 (e.g., with use of a mouse). As illustrated in FIG. 10A, the userinterface 1000 includes imagery (e.g., satellite imagery, imagerycaptured by the UAV during the first flight), and upon interaction withthe user interface 1000 to a particular portion of the user interface1000, the user interface 1000 can be updated to identify actions 1006 totake at the waypoint. Additionally, the operator 12 can specify atake-off location 1008, and optionally a landing location for the UAV 10to land at after performing the subsequent inspection of the rooftop 30.

For example, the operator 12 has interacted with the user interface 1000to specify that waypoint 2 10004 is to be inspected in a subsequentflight. The user interface 1000 therefore presents a user interfaceelement specifying actions 1006 that can be taken, which can include anautonomous descent by the UAV 12 upon traveling to the waypoint 2 10004,or an operator approved, or manually controlled, descent at the waypoint2 1004. The operator 12 can specify that the waypoint 2 10004 is to beassociated with a “Photo Survey”, which can indicate that the UAV 12 candescend towards the roof 30 upon reaching the waypoint 2 1004 (e.g.,vertically above the waypoint 2 1004). Similarly, the operator 2 canspecify that upon reaching waypoint, the UAV 12 is to hover until theoperator 12 confirms that the UAV 12 can descend (e.g., “Semi-automatedPoint Photo Survey.” The assigned waypoints and associated actions canbe provided to the UAV 12 for implementation (e.g., as described belowwith respect to FIG. 10B which is described in FIG. 1B).

Optionally a UAV can include functionality to actively determine whetherit is within line-of-sight of an operator 12. For instance, the UAV canutilize cameras to actively ensure that the operator 12 is within sight.Alternatively, the operator 12 can wear an object (e.g., a visualobject, an object that outputs particular electromagnetic radiation, andso on), and the UAV can ensure that the object is within line-of-sight.In this situation, the UAV can hold at a waypoint prior to descending ifit is not within line-of-sight of the operator 12, or can automaticallydescend if it determines it is within line-of-sight.

The waypoint information indicated by the operator 12 can be provided tothe UAV 10 for implementation (e.g., as described above). Optionally,all information associated with a waypoint (e.g., location, transitionspeed, descent speed, and so on as described above) can be associatedwith respective inputs of a command that the UAV can implement. Forinstance, the command can be associated with a particular maneuver(e.g., rooftop inspection of damaged locations), and inputs to (e.g.,parameters of) the command can be the information associated with thewaypoint. In this way, the user device can simply package theinformation indicated by the operator 12 as a command, and the UAV 10can utilize the command to perform complex maneuvers.

Optionally, the UAV 10 computer system can enforce privacy protectionsfor real-world areas outside of the property boundary geofence 14. Forinstance, the UAV 10 can determine that portions of captured imagesinclude real-world areas outside of the property boundary geofence 14.To effect this determination, the UAV 10 can utilize informationdescribing a field of view of an included camera (e.g., the field ofview can be based on a focal length of the camera, an aperture of thecamera, and so on), and position and attitude of the camera (e.g.,attitude of a gimbal that controls movement of the camera). The UAV 10can then determine a real-world field of view of the camera, and usingposition information of the UAV 10 can determine which portions ofcaptured imagery include private information (e.g., imagery outside ofthe property boundary geofence 14). Optionally, prior to storingobtained imagery, the UAV 10 can process each image to remove (e.g.,blank out) or substantially blur private portions of imagery.Optionally, the user device, or cloud system, can process each image toremove or blur private portions of imagery. For instance, as the UAV 10travels over the rooftop 30, any portions of captured imagery thatinclude private home 22 can be removed.

FIG. 1B illustrates an example of an Unmanned Aerial Vehicle (UAV) 10performing the new flight plan to perform rooftop damage inspection of aproperty 20. In the example, the operator 12 has indicated that alocation 32 on the rooftop 30 (e.g., a location in an image presented onhis/her user device that illustrates the rooftop 30) is damaged.Optionally, the selected damage type may have a selected area ofinspection, for example the user device may receive a selection of avariable radius circle (or other user affordance) over a suspecteddamaged area. The user device then determines an inspection heightdistance based on the size of the circle area. A smaller circle mayindicate that a closer height distance by the UAV to the rooftop isneeded to obtain higher resolution images, or other detailed sensordata. A corresponding GSD value based on the size of the circle may bedisplayed on the user interface. Also, the user interface of the userdevice, may allow for an inspection level for a selected damaged area.For example, different level of detailed inspection (for example, high,medium, low) may be selected. A high level of detail will identify thatthe UAV should vertically fly to a closer distance to the rooftop, thana medium level, and a medium level closer than a low level.

Different inspection levels may be needed for different damage types forneeded sensor data. Also, the level of detailed inspection maycorrespond to the sensors used for an inspection. For example, a highlevel of detail may indicate that more sensors (such as photographicimage, and thermal data is collected, whereas a low level of detail mayindicate that only a photographic image is collected). Additionally, theuser device interface may allow the user to select one or more datacollection types for the suspected damaged areas. In this case, the UAVmay have multiple payload sensor devices, or a single sensor capable ofcollecting various sensor data. For example, the user may want to selectonly thermal data to be collected for one damaged area, and onlyphotographic data to be collected for another damaged area. Thus, forthe new flight plan a waypoint is generated that includes the associatedsensor(s) that will be used or triggered to collect the particular imageor sensor data needed. Also, the user device may be configured with animage classifier, the user device can determine appropriate sensors foruse by the UAV to inspect damaged areas, and select the appropriatesensor for the particular waypoint.

The interface user affordance may represent the location to capture asingle image at the required pixel resolution (GSD), or a series ofimages. Ideally, the size of the image taken will represent nominallythe size of a physical “test square” commonly used by a human rooftopassessor. In the case of a camera that allows for electronic control,the interface for selecting the waypoint may also include other cameraconfiguration criteria, such as focal length, zoom level, aperture,time, flash, flash output level. Also, the interface may allow for theselection or input of a number of images to be taken at a waypoint. Theflight plan for each waypoint for example may be configured such thatthe camera takes two or more images of the identified damaged location.The image may be taken with the same camera settings, or the camera maybe controlled so that the multiple images have different camerasettings.

Optionally, during the initial flight the image or images obtained maybe unsuitable for the user to visually identify an area that haspossible damage. In other words, the image may not show sufficientdetail for the user to see damage to the rooftop. For example, duringthe first photo survey, the UAV may have triggered an attached camerawhen a gust of wind causes the UAV to shift, thus resulting in a blurredimage. The user device may automatically detect frames of the stitchedimage map that are not suitable for the user to identify detaileddamage. This may be done by running a process on each frame to detectits clarity or quality, or may be selected manually by the visual reviewof the operator. The user device can then mark, or designate, theparticular images as unsuitable. Also, while the UAV is in flight,sensors, such as onboard accelerometers or onboard rotational rategyros, can detect a sudden movement of the UAV. If the movement ifbeyond a threshold level, this information may be logged by the UAVcomputer system, and the image taken at that time can be identified asunsuitable. Along with the waypoints selected by the user and receivedby the user device, waypoints for the locations of the unsuitable imagesmay also be included in the flight plan. For example, for those imagesidentified as unsuitable, the user device may automatically add, oroffer to add with user confirmation, waypoints for those locationsassociated with the unsuitable images. This functionality allows for thenew flight plan to include those locations where there “may” be possibledamage, but could not be confirmed from the initial image(s) taken.During the subsequent flight, the UAV will then perform a detailedinspection of the rooftop for those locations associated with imagesthat are manually selected by the user, or determined by the userdevice, and for those images determined as unsuitable.

For a conducted flight plan, the interface of the user device maydisplay the UAV moving along the pre-planned flight path. While inflight, the UAV may transmit its geo-spatial location to the userdevice. Also, the UAV may also transmit information to the user deviceindicating that a photograph has been taken, and the coordinates of thephotograph. The UAV may also transmit a thumb-nail image of thephotograph to the user device. The interface of the user device then maydisplay a user affordance, such as an icon, or the thumb-nail image,representing a location where the image was taken. Also, the UAV may beoptionally configured to transmit in real-time the images to the userdevice. An example of a user interface to be presented on the userdevice is described below, with reference to FIG. 10B.

The UAV 10, conducting the subsequent flight plan, travels at the safedistance 16 to a location 32 of the damaged rooftop, for example byflying to a waypoint associated with the location of an identifieddamaged area. After determining that a current location of the UAV 10corresponds to the location 32 of the damaged rooftop (e.g., within athreshold tolerance, due to GPS drift), the UAV 10 descends to within athreshold distance (e.g., one meter, two meters, 5 meters, a distanceindicated by the operator 12 on the user device) to the rooftop 30 atthe location of the damaged rooftop 32. Optionally, the UAV captureslive downward-looking image frames, and compares the live capturedimages to the previously-generated stitched mosaic, or compared to a“central” photo. Comparing between the live image frames and mosaic, or“central” photo involves extracting visual features in both images andrunning (e.g., executing) a feature matching algorithm, such as RANSAC(e.g., Random sample consensus), across them, which can be used toestimate the relative pose (position and orientation) of the UAV camerawith respect to the geo-rectified stitched mosaic, or the “central”photo. The relative pose is then used determine the location of the UAV(and the UAV's location with respect to the denoted damaged rooftop) insituations when other localization devices (e.g. GPS) are unavailable orunreliable. As the UAV 10 descends, it activates sensors included withthe UAV 10, and captures real-world information describing the damagedrooftop 32. The UAV 10 can reach a threshold distance above the rooftop30 prior to activating sensors (e.g., 5 meters, 10 meters, the distanceindicated by the GSD), or the UAV 10 can activate sensors periodicallystarting from the safe distance 16.

Additionally, as described above the UAV 10 can travel to a waypoint(e.g., location 32), and hover at the waypoint until receivinginformation from the operator 12 indicating the UAV 10 can safelydescend. Optionally, the operator 12 can view a user interface 1050 thatincludes each waypoint the UAV 10 is to travel to, and specifies apresent location of the UAV 10 (e.g., the UAV 10 can provide locationinformation while flying, to indicate a progress of the UAV 10performing the inspection of the rooftop 30).

FIG. 10B illustrates an example user interface 1050 showing arepresentation 1056 of the UAV 10 implementing a flight plan. Asdescribed above, an operator 12, or other user, can monitor the UAV 10moving along a particular flight path. As illustrated in FIG. 10B, therepresentation 1056 of the UAV 10 is traveling from a first waypoint1060 (e.g., a safe take-off location) to a second waypoint 1054 adjacentto multiple inspection waypoints (e.g., a waypoint at which the UAV 10is to descend and perform an inspection for damage, such as inspectionwaypoints A-D). The user interface 1050 can include imagery (e.g.,stitched imagery) obtained during the initial flight plan (e.g.,described above with respect to FIG. 1A), or can include satelliteimagery of the property 20. As described above, in FIG. 1A, therepresentation 1056 of a UAV can optionally be bounded by aquadrilateral 1058 representing a field of view of a sensor of the UAV10 (e.g., a camera, with the field of view being determined by thealtitude, location, and attitude information of the UAV 10). Optionally,the quadrilateral 1058 can be presented when the UAV 10 is conductingthe initial flight plan, and not when the UAV 10 is implementing theflight plan to perform rooftop inspection. In this way, the operator 12can quickly determine, in the initial flight plan, that the UAV 10captured imagery of the entirety of the rooftop 30.

Optionally, the quadrilateral 1058 can be presented while the UAV 10 isimplementing the flight plan to perform the rooftop inspection, and oncethe UAV 10 reaches an inspection waypoint (e.g., inspection waypoint A)and descends, the quadrilateral 1058 can decrease in size during thedescent (e.g., since the UAVs 10 field of view would decrease). Once theUAV 10 reaches within the threshold distance of the rooftop (e.g., basedon the user affordance described above), the quadrilateral 1058 canhighlight, flash, or otherwise alert the operator 12 that the field ofview includes acceptable detail of the rooftop 30.

Additionally, optionally upon the UAV 10 navigating to an inspectionwaypoint (e.g., waypoint A), the user interface 1050 can be updated toindicate that the UAV 10 is waiting for the operator's 12 confirmationto descend. The operator 12 can then utilize the user interface 1050 toeither provide information to the UAV 10 indicating the UAV 10 cansafely descend, or the operator 12 can provide manual control of the UAV10 (e.g., control a descend throttle). The manual control can,optionally, only allow the operator 12 to control a vertical descenttowards the rooftop 30. For instance, any manual commands associatedwith pitch, roll, and yaw, will be discarded by the operator's 12 userdevice, or optionally by the UAV 10. Optionally, the operator 12 canoverride the discarded commands, and manually control pitch, roll, yaw,and so on.

As the UAV 10 descends, the UAV 10 uses an included location sensor(e.g., a GPS receiver) to maintain the UAV's 10 location in a verticaldescent towards the location 32 of the damaged rooftop. Since anymovement not purely in the vertical axis can cause the sensors to obtainreal-world information (images) not specifically of the damaged portion32, the UAV 10 optionally ensures that it moves substantially in thevertical direction. For instance, the UAV 10 can utilize sensors thatmeasure movement (e.g., an Inertial Navigation System (INS) thatmeasures position and attitude information, or using visual odometrysystem to measure position via digital images, such as stored images orimages received from the user device) to detect whether the UAV ismoving horizontally, and if so corrects the movement. Additionally, theUAV 10 can optionally ensure that the center of obtained images of thedamaged rooftop 32 (e.g., as the UAV 10 descends) correspond to thecenter of the image obtained during the initial flight (e.g., describedin FIG. 1A), that was indicated by the operator 12 as being damaged. Forinstance, as described above, the operator 12 can select an area of therooftop 30 that he/she determines to be damaged or likely to be damaged.Since the UAV 10 therefore has image data of the damaged area 32, theUAV 10 can utilize one or more visual classifiers and/or computer visionalgorithms, to determine that the center of the images being obtainedduring descent correspond to the image data of the damaged area 32.Optionally, the UAV 10 can have information describing a 3D model of theproperty 20.

To ensure that the UAV 10 descends vertically, the UAV 10 can utilize asensor that measures distance (e.g., a LiDAR sensor, a Leddar sensor),and utilizing information describing a speed of the UAV's 10 descent(e.g., the UAV 10 can determine velocity from distance traveled within aperiod of time, and in particular, can determine speed of a downwardvertical distance traveled within a period of time), the UAV 10 candetermine whether the measured distance information comports with thespeed of the UAV 10. For instance, if the rooftop 30 is at a slope, ifthe UAV 10 drifts in one direction, the distance might increase, ordecrease, due to the UAVs 10 movement, and not just due to the speed ofdescent. Therefore, the UAV 10 can determine that it moved horizontally,and using the 3D model can determine and apply an appropriatecorrection. Similarly, the UAV 10 can use imagery obtained by cameras tocompare its location to information describing the 3D model of theproperty (e.g., the UAV 10 can determine that it is located at aparticular three-dimensional coordinate within the 3D model space).

The real-world information 16 can be gathered according to one or moretriggers as the UAV 10 descends. For instance, the UAV can selectivelyactivate one or more of the included sensors, periodically (e.g., every3 seconds, 10 seconds, operator 12 indicated amount of time), based on adistance descended (e.g., every 10 centimeters, every ¼ meter, everymeter, operator 12 indicated distance, and so on, where an altimeter(e.g., a laser, barometric, or radio altimeter) may be used to determinealtitude and a change in altitude), in response to an event, etc. Forexample, if a first sensor senses a first event, a second sensor may beactivated in response. Additionally, the triggers can depend oncapabilities of the UAV and included sensors, such as a focal length ofthe camera (e.g., a shorter focal length can provide a wider angle ofview, enabling less pictures to be taken), an aperture of the camera(e.g., a wider aperture can enable the camera to obtain pictures withless noise, by utilizing a lower ISO, but have shallower depth of fieldand potentially require more images), and so on.

The UAV 10 descends until reaching the threshold distance from therooftop 30, then ascends back to the safe distance 16. The UAV 10 thentravels to one or more other locations identified by the operator 12 asbeing damaged, and travels to a safe landing location. The real-worldinformation obtained by the UAV 10 is provided to the user device of theuser 12. For instance, the UAV 10 can provide the information wirelessly(e.g., using a BLUETOOTH connection, a Wi-Fi connection, a near fieldcommunication connection, and so on) and/or using a wired connectionwith the user device. Additionally, the UAV 10 can provide thereal-world information as it obtains it (e.g., in substantiallyreal-time).

The UAV 10, the user device of the operator, or the cloud system, canoptionally automatically classify (e.g., using visual classifiers,computer vision algorithms) damage seen in the real-world information.As will be described, the rooftop damage can be characterized (e.g., oneor more labels can be assigned to the damage), such as hail damage, nodamage, wind damage, poor imagery, and so on. Optionally, real-worldinformation of the property 20 in a prior period of time (e.g., prior todamage) can be used as a reference point when determining damage. Thatis, the UAV 10, user device, or cloud system, can compare the presentobtained real-world information to the prior real-world information as abaseline. For example, a historical record of the thermal images of arooftop may be compared to a new thermal image of the rooftop to aid adetermination as to whether the rooftop has been damaged (e.g., bydetermining if the rooftop is radiating significantly more heat thanunder historically similar weather conditions). Additionally, thereal-world information can be used to train, or update, the one or morevisual classifiers (e.g., a reviewing user can determine a correct labelfor the damage, and the visual classifiers can be updated to incorporatethe correctly labeled damage).

While the UAV computer system autopilot module is navigating the UAV forthe inspection, certain aspects of the flight pattern may be controlledby the operator's user device. For example, while the UAV is performingan inspection of a rooftop of a structure, the UAV would be ascendingand/or descending and obtain sensor information describing thestructure, for example triggering a camera to obtain digital images. Theflight plan or pattern may be configured such that for a particularinspection location a vertical ascent/descent rate, UAV altitude,horizontal UAV rotation, payload gimble, payload direction, or trigger apayload sensor may be controlled by the operator. The user device mayhave a physical control such as a toggle or joystick, or a userinterface control, that allows the user to control verticalascent/descent rate, UAV altitude, UAV attitude, horizontal UAVrotation, payload gimble, payload direction, or trigger a payload sensorwhile conducting the inspection. For example, the UAV may navigation viaautopilot to a position over an inspection location, and the operatorthen can provide input to the user device, and the user device maytransmit a signal or information corresponding to the user input, to theUAV (via radio communication) to control the vertical ascent/descentrate, UAV altitude, UAV attitude, horizontal UAV rotation, payloadgimble, or payload direction, or trigger a payload sensor. Thisparticular mode allows for partial auto-pilot control, and manualoperator control of the UAV. In this case, the UAV automatically viaautopilot moves from inspection location to inspection location, but theoperator has the ability to control the vertical ascent/descent rate,UAV altitude, UAV attitude, horizontal UAV rotation, payload gimble, orpayload direction, or trigger a payload sensor. However, even though theoperator may control the vertical ascent/descent rate, the UAV still mayenforce the highest altitude, and the lowest altitude the UAV may flyto, or closest distance allowed to the structure. Additionally, theoperator may choose to hold the position of the UAV, and manuallytrigger one or more sensors of the UAV. The user device may receive aninput indicating that the UAV should move to the next inspectionlocation (or could receive an input to move to a previous inspectionlocation). In this case, the UAV will then resume autopilot mode andmove over the next inspection location to perform an inspection of therooftop for possible damage. The inspection may continue in an autopilotmode, or may again be partially manually controlled by the operatorusing the user device. Additionally, the UAV may receive a command fromthe user device to nudge the UAV in a particular direction. In thiscase, the control input of the user device, allows for sending a commandto the UAV to move slightly, for example between 0.1 to 3 meters, in aparticular direction (in an x, y, or z axis, or diagonally). Theparticular distance can be predetermined, or be variable based on theproximity to the structure. Nudging the UAV, allows the operator to movethe UAV away from the structure if the operator sees that the UAV flyingtoo close to the rooftop. The nudge command may be provided any time tothe UAV while it is operating in an auto-piloted mode. However, the UAVshould still enforce geofence boundaries and not allow a nudge to causethe UAV to move beyond across a geofence boundary envelope.

FIG. 2 illustrates a block diagram of example systems utilized inperforming a rooftop inspection of a property. The block diagramincludes a user device 110 in communication with an Unmanned AerialVehicle (UAV) 100 and a cloud system 120 (e.g., a system of one or morecomputers in communication with the user device 110 over a network, suchas the Internet). Additionally, the UAV 100 can be optionally incommunication with the cloud system 120 (e.g., over a network, such asthe Internet, or through an intermediate system). As described above,the cloud system 120 can determine job information 126 describing one ormore tasks to perform inspections of one or more properties, and the UAV100 can perform inspections of the one or more properties.

The cloud system 120 includes a job determination engine 122 that canreceive, or obtain, information describing jobs, and determine jobinformation 126. The job determination engine 122 can generateinteractive user interfaces (e.g., web pages to be rendered by a userdevice) for presentation on a user device (e.g., the user device 110). Auser of the user device (e.g., user device 110) can provide informationassociated with a particular job in the interactive user interfaces.

For instance, the user can enter an address of the property to beinspected, and the job determination engine 122 can obtain informationdescribing the property. The information can include, propertyboundaries of the property (e.g., from a database, or system that storesor can access property boundary information), geo-rectified imagery(e.g., satellite imagery) of the property, and so on. The jobdetermination engine 122 can determine a property geofence envelope forthe UAV 100 to enforce (e.g., the UAV 100 can be required to remainwithin or substantially within the property boundaries of the property).

Similarly, the job determination engine 122 can receive informationdescribing the rooftop to inspect at the property. The cloud system 120can generate user interface data that includes imagery of the property,and the user can indicate boundaries of the rooftop. For instance, asdescribed above, the user can select corners of a polygon thatencompasses the rooftop (e.g., corners of the rooftop).

After determining a geofence envelope, the job determination engine 122can receive information describing an expected type of damage, andinformation relevant to configuration information of the UAV 100 (e.g.,information describing a type of UAV, sensors included in the UAV, andgeneral functionality that can be performed by the UAV 100). Forinstance, the job determination engine 122 can receive informationidentifying that hail damage is expected, or is to be looked for, andcan determine that a UAV which includes particular sensors, and specificvisual classifiers to identify hail damage, is needed (e.g., a heatand/or thermal imaging sensors, specific visual classifiers that candiscriminate hail damage from other types of damage, wind damage, raindamage, and so on).

The job determination engine 122 can receive a time that the job is tobe performed (e.g., a particular day, a particular time at a particularday, a range of times, and so on). The job determination engine 122 canthen determine an availability of UAVs and/or operators at the receivedtime(s). Additionally, the job determination engine 122 can filter theavailable UAVs according to determined configuration information (e.g.,as described above). Optionally, the job determination engine 122 canaccess weather information associated with the received time(s), anddetermine an optimal time or range of times for the job to be performed.For instance, a UAV that includes particular sensors (e.g.,electro-optic sensors) can obtain better real-world information atparticular times of day (e.g., at noon on a sunny day can provide betterimagery by maximizing image contrast and minimizing the effects ofshadows).

The job determination engine 122 can then provide the determined jobinformation 126 to the user device 110, and optionally directly to theUAV 100. For instance, the UAV 100 can be located in an area with amultitude of UAVs, and the cloud system can select the UAV 100 (e.g.,based off configuration information, availability, and so on).Optionally, the user, via the user device 110, can override the cloudsystem UAV selection, and can instruct that the job be provided to a UAVselected by the user. The UAV 100 can then receive the job information126 (e.g., over a wired connection, or over a wireless connection suchas Wi-Fi, BLUETOOTH, and so on).

The user device 110 includes an application engine 112 that can receivethe job information 126, and can generate user interface data describingthe job information 126. An operator of the UAV 100 can travel to theproperty identified in the job information 126 with the UAV 100, andview information describing the job information 126. Optionally, theapplication engine 112 can receive modifications, from the operator, tothe job information 126, including updates to the rooftop boundary, andso on. Optionally, the application engine 112 can allow particularmodifications (e.g., rooftop boundary), but not other modifications(e.g., the property boundary cannot be modified). The application engine112 can receive information to effect the job, including a safe altitude(e.g., as illustrated in FIGS. 1A-1B), and safe take-off/landinglocations (e.g., the application engine 112 can display imagery of theproperty, and the operator can indicate the safe locations).

The application engine 112 can provide flight information 114 to the UAV100, which is information sufficient to effect a safe inspection of therooftop according to the job information 126. For instance, the flightinformation 114 can include a geofence envelope determined from theproperty boundary, the safe altitude, boundaries of the rooftop, and soon. The flight information 114 can further include a flight plan for theUAV 100 to follow. As described above, the UAV 100 can travel over theproperty according to particular flight patterns that can depend on therooftop boundaries, and configuration information of the UAV 100. Forexample, the application engine 112 can determine that the UAV 100 is tofly in a particular zig-zag pattern (e.g., as described above), based onconfiguration information of the UAV 100 and the inputted safe altitude.That is, the application engine 112 can determine that based on theparticular focal length, sensor resolution, and so on, of a cameraincluded in the UAV 100, that the distance between each leg of thezig-zag pattern is to be a particular distance apart (e.g., a leg can betraveling along a first axis of the rooftop, and a subsequent leg can betraveling along the first axis in the opposite direction shifted by adistance in the orthogonal axis). In this way, the application engine112 can determine that there will not be holes in the camera's coverage,such that the images of the rooftop can be stitched together (e.g., eachimage will contain enough visual information to identify subsequent andprior legs of the zig-zag pattern), and that the entire rooftop can beimaged at a particular resolution (e.g., a particular number ofpixels/distance of the rooftop). Determining a flight pattern isdescribed in more detail below, with respect to FIG. 4.

The UAV can receive the flight information 114 from the user device 110(e.g., over a wired connection, or a wireless connection). The UAV 100includes a UAV application engine 102 that can effect the flightinformation 114 that identifies the job to be performed. As illustratedin FIG. 1A-1B, the UAV 100 can ascend to the safe altitude (e.g.,identified in the flight information 114), travel over the rooftopaccording to the flight pattern, and activate sensors included in theUAV 100 to obtain real-world information describing the rooftop. Afterobtaining the real-world information, the UAV 100 can provide sensorinformation 106 to the user device 110. The user device 110 can combinethe received sensor information 106 (e.g., stitch together images of therooftop, generate a 3D model of the property, and so on), and theoperator can indicate, on the user device 110, areas of the rooftop thatare damaged. The UAV 100 can receive information identifying the areasfrom the user device 110, and can perform detailed inspections of thedamaged areas (e.g., as described in FIG. 1B). The sensor information106 associated with the detailed inspection can be provided to the userdevice 110.

To control the UAV 100, the UAV 100 includes a flight control engine 104that can manage the motors, rotors, propellers, and so on, included inthe UAV 100 to effect the flight information 114. Optionally, the UAVapplication engine 102 can provide commands (e.g., high level commands)to the flight control engine 104, which can interpret the commands toperform the inspection. For instance, the UAV application engine 102 canindicate that the UAV 100 is to descend at a particular locationidentified as being damaged, and the flight control engine 104 canensure that the UAV 100 descends in a substantially vertical direction.

After receiving sensor information 106 associated with the detailedinspection of damaged areas, the user device 110 can generate one ormore interactive reports 116 describing the damage. A report 116 can bean interactive document (e.g., a web page) that can be provided forpresentation on the user device 110 (e.g., in a web browser), or to anoutside user device. The report 116 can include a graphicalrepresentation of the property (e.g., a dimensioned, graphical map ofthe rooftop) with damaged areas identified (e.g., highlighted). Theoperator can indicate types of damage identified, and the report 116 candescribe the types of damage, an area of each damaged area, and canprovide a reviewing user of the report 116 with access to the raw sensorinformation 106 for each damaged area. Optionally, the user device 110or cloud system 120 can determine types of damage from one or morevisual classifiers that can operate on the received sensor information106.

The cloud system 120 is in communication with a generated reports 129database, that can receive the report 116 and store the report 116 withinformation describing the report (e.g., information describing theproperty and rooftop, configuration information of the UAV 100, timeinformation associated with the inspection, and so on). Optionally, thecloud system 120 is in communication with a classifier information 128database (e.g., one or more databases, or a storage subsystem) that canstore information describing one or more visual classifiers (e.g.,information utilized by one or more machine learning models, such assupport vector machines, k-means clustering, neural networks, and soon). A reviewing user can review the report 116, and correctly classifythe types of damage identified in the sensor information 106. Thiscorrectly classified information can be used to update the one or morevisual classifiers.

FIG. 3 is a flowchart of an example process 300 for determininginformation describing a job for a UAV to perform an inspection of arooftop. For convenience, the process 300 will be described as beingperformed by a system of one or more computers (e.g., the cloud system120).

The system obtains information describing a job to perform an inspectionof a rooftop (block 302). As described above, the system can generateuser interfaces for presentation on a user device of a user that enablethe user to enter information associated with a job to perform aninspection. For instance, a severe weather event can cause the user(e.g., an employee of an insurance company) to want to quickly inspectproperties included in affected areas. The system can receive a requestfrom the user regarding an inspection, and can provide user interfaces(e.g., one or more web pages) for presentation on the user device of theuser, or the system can be associated with a display (e.g., over a wiredor wireless connection), and can provide user interfaces forpresentation on the display.

The system receives information describing a property to be inspected.For instance, the user can enter an address associated with theproperty, and the system can obtain information describing the property.For instance, the system can obtain boundaries of the property (e.g.,from commercial, or governmental, databases or systems), or the systemcan obtain geo-rectified imagery of the property (e.g., satellite, ormapping, imagery), and provide the imagery for presentation on the userdevice of the user. The system can receive selections of the boundaryfrom the user (e.g., the user can trace the property boundaries), or thesystem can determine the boundaries from the imagery (e.g., the systemcan utilize computer vision techniques to identify the boundaries). Thesystem stores information describing the property boundaries.

The system receives information describing the rooftop included in theproperty boundaries to be inspected. The system can determine theboundaries from the imagery of the property (e.g., the system canidentify contours of the rooftop using edge detection techniques, orusing k-means clustering and machine learning algorithms to identify theboundaries as described above). Additionally, the system can provideimagery of the property for presentation on the user device, and theuser can indicate boundaries of the rooftop. For instance, the user canselect (e.g., the user can click on corners of the rooftop with a mouse,with his/her finger, and so on) the boundaries of the rooftop.Optionally, the user can trace the boundaries of the rooftop, and thesystem can correlate the tracing to the image of the rooftop, toidentify a correct boundary (e.g., a contour of the rooftop). Forinstance, a particular rooftop can be circular in shape, and the usercan trace a circle corresponding to the rooftop. The system storesinformation describing the rooftop (e.g., GPS coordinates of theboundaries, or information describing location and shape information ofthe contour of the rooftop).

The system can optionally receive a starting time of the job to performthe inspection, or a particular range of times that the user wouldprefer. As described above, the system can obtain weather information,and determine an optimum starting time (or range of starting times) ofthe job. The system can provide recommendations regarding starting timesfor presentation to the user, and the user can select from among therecommendations, or the user can select a different time. The system canobtain indications of schedules of other jobs, and schedules ofoperators that can perform the job. The system can provide informationto the user indicating operators that are available to travel to theproperty, and perform the job in concert with a UAV at the selectedtime. Additionally, the system can modify the selected time according toother jobs that are being performed within a threshold distance of theidentified property. For instance, the system can determine that adifferent job to inspect an adjacent property is to be performed at aparticular time, and can recommend that to the user that operator andUAV performing the different job also perform the entered job at thesame time (e.g., subsequent to the different job). In this way, thesystem can batch multiple jobs together to be efficiently performed by asame UAV and operator.

As an example of utilizing weather information, the system can determinewhether weather will negatively affect the inspection. For instance, thesystem can determine that the weather will be cloudy, rainy, and so on,and that one or more of the sensors included in the UAV will benegatively affected or if the flying conditions will be unsafe.Furthermore, the system can determine locations of the sun during theinspection, and based on the locations, can determine whether the sunwill be pointed at the one or more of the sensors of the UAV (e.g., thebright light of the sun can cause clipping in the image) based on thestart time of the inspection and location information of the property.The system can recommend alternate times, or recommend a particular UAVto perform the inspection (e.g., a UAV with filters to reduce brightlight).

The system generates a property boundary geofence (block 304). Thesystem determines a property boundary geofence for the UAV to enforce,ensuring that the UAV is to remain with the property boundary of theproperty to be inspected. The system can store location information ofthe geofence, which can include a 2D envelope (e.g., limiting the UAV toa 2D area in the real-world) or a 3D envelope (e.g., limiting the UAV toa 3D volume in the real-world.). As described above, with reference toblock 302, the system can obtain information describing a propertyboundary. For instance, the property boundary information can include aspeciation of precise location coordinates (e.g., GNSS coordinates) thatdescribe one or more locations on a peripheral of the boundary.Additionally, the property boundary information can include curveinformation sufficient to describe the boundary, for instance Beziercurves. The property boundary can be utilized to define the propertyboundary geofence for the UAV to enforce.

The system optionally determines a safe take-off location and a safelanding location (block 306). In some implementations, the system candetermine a take-off location (e.g., a location from which the UAV is tobegin the job), and a landing location (e.g., a location at which theUAV is to complete the job).

As described in block 302, the system can obtain imagery (e.g.,satellite imagery) of the property. The system can analyze the obtainedimagery to determine a safe take-off location that is suitable for theUAV to initiate the job. To determine a suitable location, the systemcan utilize computer vision techniques to identify features that areknown, by the system, to indicate a suitable take-off location. Forinstance, the system can determine whether the obtained imagery includesa sidewalk, a driveway, a roadway (e.g., a street), or other openclearing. Optionally, the system can obtain location informationindicating boundary information of sidewalks, roadways, and so on (e.g.,from a government or commercial database), and can utilize the boundaryinformation to identify sidewalks, roadways, and so on, in the obtainedimagery. For instance, the system can determine, from the boundaryinformation, locations in the obtained imagery that correspond tosidewalks, roadways, and so on. The system can then analyze thelocations to ensure that no obstructions exist (e.g., fire-hydrants,light posts, and so on).

Furthermore, the system can utilize color information included in theobtained imagery to better determine whether particular featurescorrespond to driveways, roadways, sidewalks. That is, a substantiallyray color can likely correspond to a driveway, whereas a substantiallygreen color can correspond to foliage, trees, and so on. Optionally, thesystem can determine a measure of central tendency of a color of alocation, and a variance from the measure of central tendency. In thisway, the system can determine whether a driveway, or other cement objectknown to be suitable, is substantially gray, and if the driveway has avariance greater than a threshold, the system can determine the drivewayis unsuitable.

Similarly the system can determine one or more locations that are notsuitable as a safe take-off location. For instance, the system canidentify features that are known to indicate that a location is notsuitable (e.g., features of a house, a fence, a pool, a tree, a bush,within a threshold distance of a power line, and so on).

The system can present information to the user indicating locations thathave been determined to be suitable, and optionally locations that havebeen determined to not be suitable. The user can then select a suitablelocation, and the system can designate the location (e.g., GNSScoordinates associated with the location) as the safe take-off location.Optionally, the system can receive a selection of a particular locationthe user prefers, and the system can determine whether the location issuitable (e.g., likely suitable greater than a threshold), or unsuitable(e.g., likely unsuitable at greater than a threshold). Upon adetermination that the location is unsuitable, the system can presentuser interface data indicating the location is unsuitable and can blockthe location from being selected. Optionally, the user can override thesystem's determination, and the system can designate the location as thesafe take-off location.

Additionally, the system can perform the above analysis and processes todetermine a safe landing location. The safe landing location can bedesignated, and utilized by the UAV when performing the job. Optionally,the system can designate multiple safe landing locations, such that if acontingency situation is determined (e.g., the UAV is running out ofbattery charge, liquid fuel, and so on), the UAV can safely land at anearest safe landing location.

The system optionally selects a UAV to perform the job (block 308). Asdescribed above, the system can optionally receive informationdescribing the types of damage of interest. For instance, the system cangenerate user interface data that includes selectable options describingdifferent types of damage for selection by the user.

The system can determine configuration information of a UAV that isbeneficial to identifying the selected type of damage. For instance, todetermine hail damage the UAV can benefit from a heat sensor or thermalimager (e.g., older hail damage has higher heat loss than newer haildamage). Additionally, a given type of damage may benefit from aninfra-red sensor, an ultra-violet sensor, a sensor that can measureradiation, a sensor that can detect chemical leaks, a camera with alonger focal length (e.g., enabling the UAV to take detailed imageswithout getting too close to the property), and so on.

The system can then obtain configuration information of UAVs that areavailable to be utilized during the selected time (e.g., described abovein block 302). The system can identify the UAVs that include thedetermined configuration information, and select a UAV. Additionally,the system can provide information to the user describing the determinedconfiguration information, and the user can specify the configurationinformation when an operator selects a UAV to bring to the property.

The system provides information describing the job to a UAV and/or userdevice in communication with the UAV (block 310). The system can provideinformation describing the job to the UAV (e.g., the selected UAVdescribed above), and the UAV can be taken by an operator to theproperty. Similarly, the job information can be provided to a userdevice of the operator, and operator can modify information, and providethe modified information to the UAV.

FIG. 4 is a flowchart of an example process for determining flightinformation to perform a job. For convenience, the process 400 will bedescribed as being performed by a user device of one or more processors(e.g., the user device 110). Optionally, the process 400, or particularblocks or features of the process 400, can be implemented by the cloudsystem 120. As a non-limiting example, the cloud system 120 can performblock 408.

The user device receives job information (block 402). As describedabove, the system can determine job information (e.g., from informationreceived from a user of the system), and can provide the job informationto the user device.

The user device presents user interface data describing the job (block404). The user device generates user interface data to describeinformation entered by the user that requested the inspection beperformed (e.g., described in FIG. 3). For instance, the user device cangenerate one or more interactive documents (e.g., web pages), thatinclude summary data describing the inspection.

The user device can present a graphical representation of the property(e.g., satellite or mapping imagery) with the property boundaryidentified (e.g., highlighted), and the rooftop identified (e.g., therooftop boundary can be highlighted, the rooftop can be colored aparticular color, the rooftop can be shaded, a determined flight plan toinspect the rooftop can be illustrated, and so on). Optionally, the userdevice can receive modifications to the presented information. Forinstance, the operator can modify the rooftop boundary (e.g., trace anupdated boundary, indicate updated corners of a polygon that encompassesthe rooftop and so on.) Additionally, the operator can indicate that anobstruction exists in a particular location on the property (e.g., alarge tree, antenna, or air conditioning unit can cover a portion of therooftop). Optionally, the operator can take imagery of the obstructionand its location in the property, and the user device can determineboundaries of the obstruction. The user device can store informationdescribing the obstruction, which can be used by a UAV when traveling(e.g., flying).

The user device receives additional flight information (block 406). Theuser device receives information specific to the property, including asafe altitude above which the UAV can travel freely without encounteringobstructions within the property boundary. The operator can visuallyinspect the property, and determine the safe altitude (e.g., estimate asafe altitude). Optionally, the operator can take an image of theproperty (e.g., the operator can hold a camera and take a picture), andusing height information of the property (e.g., obtained from a propertydatabase), the user device can determine an estimated height ofobstructions (e.g., trees) within the property boundary. Additionally,the user device can receive location information (GPS coordinates)identifying a safe take-off location, and a safe landing location.

The user device, or optionally cloud system 120, determines a flightpattern for the UAV to follow (block 408). As described above, withrespect to FIG. 1A, the UAV can follow a particular flight path that canbe based off a boundary of the rooftop (e.g., an area of the rooftop,and shape information of the rooftop) and configuration information ofthe UAV (e.g., whether the UAV is a fixed-wing aircraft or a multi-rotoraircraft that can make sharp turns). Block 408 can optionally beperformed by the cloud system (e.g., cloud system 120) or the UAV (e.g.,UAV 100). The flight pattern can be associated with real-worldcoordinates (e.g., GPS coordinates) for the UAV to follow, or the UAVcan follow the flight path using imagery obtained of the rooftop whileflying along with location information of the UAV.

For a multi-rotor aircraft (e.g., a quadcopter, and so on), the userdevice can determine that a zig-zag pattern is to be used. As describedin FIG. 1A, the zig-zag pattern describes a pattern in which the UAVflies parallel to a first axis associated with the rooftop boundary,turns towards an orthogonal axis for a particular distance, and thenflies parallel to the first axis in the opposite direction. Each flightalong the first axis can describe a leg of the zig-zag flight pattern.The user device can define the particular distance to travel along theorthogonal axis such that sensor information obtained by the UAV inflight includes overlapping real-world information (e.g., imagery) froma first leg and a subsequent adjacent second leg (e.g., when the UAVtravels the opposite direction along the first axis shifted in theorthogonal axis by the particular distance). The particular distance canbe based off configuration information of the UAV, such as a focallength of an included camera. For instance, a camera with a long focallength will need a comparatively shorter distance between legs than awide-angle lens, since the camera can see less of the rooftop in eachimage. Therefore, the user device can determine the particular distance,and a number of legs that will cause the UAV to obtain sensorinformation of the entire rooftop.

The flight pattern can include waypoints that describe the determinedflight pattern. For instance, the waypoints can indicate lengths alongthe first axis, and a distance from the first axis to the second axis(e.g., distance of each leg, and distance between legs, as describedabove). Optionally, the UAV can execute an application (e.g., a userapplication) that can receive the waypoints, and utilize the waypointsto effect the flight pattern. That is, the user device, or cloud system120, can package the flight pattern into one or more waypoints that areunderstandable, to the application, as being associated with the zig-zagpattern.

Additionally, optionally the user device can receive an indication of aground sampling distance (e.g., a number of image pixels/distance).Using the configuration information (e.g., a number of pixels of asensor included in a camera of the UAV), the user device can determine aminimum altitude for the UAV to fly at. If the minimum altitude isgreater than the safe altitude, the user device can prefer that the UAVfly closer to the minimum altitude. In this way, since the UAV is flyinghigher and can capture a greater area of the rooftop in each image, thenumber of legs in the zig-zag pattern can be reduced.

The user device provides the flight information to the UAV (block 410).The user device provides sufficient information to the UAV to enable theUAV to perform the inspection. For instance, the user device can providethe safe altitude, the flight pattern, and so on. After receipt of theflight information, the user device can provide a request, orinstruction, to the UAV to begin the inspection. That is, the userdevice can execute an application (e.g., an “app” downloaded from anapplication store, or otherwise installed on the user device) that is incommunication with the UAV.

As described above, with reference to FIG. 1A-1B, the UAV can perform aninitial inspection to obtain sensor information of the rooftop (e.g.,images). After damaged areas of the rooftop are identified, a subsequentinspection can be performed to obtain detailed sensor information ofeach identified damaged area.

FIG. 5 is a flowchart of an example process to perform an initialinspection of the rooftop. For convenience, the process 500 will bedescribed as being performed by a system of one or more processors(e.g., the UAV 100).

The UAV receives flight information describing the inspection job to beperformed (block 502). As described above, the UAV receives flightinformation sufficient to enable to the UAV to perform the inspection.

The UAV ascends to the safe altitude from the safe take-off location(block 504). The UAV can be moved (e.g., by the operator) to the safetake-off location, and upon receipt of a request to begin the initialinspection, the UAV can ascend to the safe altitude. Optionally, the UAVcan immediately ascend to the safe altitude and assume that once itreceives the request to begin the initial inspection, it is located at asafe take-off location.

Once the UAV reaches the safe altitude, it travels at the safe altitudeto an initial location over the rooftop. The flight pattern informationcan indicate a starting point to perform the inspection, or the UAV candetermine a closest point of the rooftop, travel to the closest point,and then begin the flight pattern based on the closest point.

The UAV performs the initial inspection (block 506). The UAV travelsalong the determined flight path (e.g., determined by the user device,cloud system, or by the UAV), and activates sensors included in the UAValong the flight path. The UAV can activate the sensors (e.g., obtainimages) periodically in time (e.g., every half a second, every second),or periodically in distance (e.g., every meter, every two meters). Theperiodic schedule can depend on configuration information of the sensorsincluded in the UAV, for instance, a camera with a longer focal lengthand a smaller field of view can require a greater number of images to betaken since each image includes a comparatively smaller area of therooftop than a shorter focal length.

The obtained real-world information from the sensors can be provided(e.g., substantially in real-time) to the user device of the operator.Optionally, the user device can provide directions to the UAV in flight.For instance, the operator can indicate that a portion of the real-worldinformation obtained by the UAV was of poor quality (e.g., out of focusimage) or was not obtained. The UAV can then receive informationdescribing a location of the rooftop that is to be flown over again toobtain updated real-world information.

The UAV travels to the safe landing location after inspecting therooftop (block 508). Upon completion of the flight path, the UAV travelsto safe landing location at the safe altitude, and descends back to theground, or hovers at the safe landing location. Optionally, thereal-world information obtained by the UAV can be provided to the userdevice along with metadata for the information (e.g., position/attitudeinformation of the UAV for each image, gimbal attitude of a camera whentaking an image, and so on).

The user device, or cloud system, generates a geo-rectified image of theobtained real-world information and metadata obtained from the UAV. Theuser device, or cloud system, can utilize photogrammetry software tostitch the images together to generate a geo-rectified image of therooftop. Optionally, the user device, or cloud system, can generate a 3Dmodel of the rooftop and property. As described above, the UAV canutilize a wide-angle lens to obtain images of the sides of the property(e.g., a side of a home), and using the obtained imagery, a 3D model canbe generated.

The user device presents the geo-rectified image of the rooftop, and/orpresents the generated 3D model in one or more user interfaces. Theoperator indicates areas of the rooftop that he/she determines to bedamaged, and the user device stores location information (e.g., GPScoordinates) of the damaged areas. Alternatively or in addition, theuser device, or cloud system, can optionally automatically determinedamaged areas (e.g., as will be described below). The user device thenprovides location information of the damaged areas to the UAV.Optionally, the operator can swap out one or more sensors included inthe UAV for the subsequent inspection. For instance, the operator canremove a camera included in the UAV, and include a camera with acomparatively longer focal length.

FIG. 6 is a flowchart of an example process 600 for performing asubsequent inspection of the rooftop. For convenience, the process 600will be described as being performed by a system of one or moreprocessors (e.g., the UAV 100).

The UAV obtains location information of damaged areas of the rooftop(block 602). As described above, the operator can indicate damaged areasof the rooftop, and the user device can provide location informationidentifying the damaged areas.

The UAV ascends to the safe altitude, and performs an inspection of aninitial damaged area (block 604). As illustrated in FIG. 1B, the UAVtravels to an initial damaged area (e.g., a damaged area closest to theUAV), and descends vertically towards to the rooftop at the initialdamaged area.

As the UAV descends, the UAV activates the sensors included in the UAVto obtain real-world information describing the damaged area. Asdescribed above, the UAV can activate the sensors after being within athreshold distance above the rooftop (e.g., determined using a LiDARsensor, a barometric sensor, and so on), and can obtain real-worldinformation periodically until reaching a threshold distance above therooftop (e.g., a meter, two meters).

As described with respect to FIG. 1B, as the UAV descends it ensuresthat it is descending vertically to maintain the damaged area as thefocus of the included sensors. That is, a GPS receiver, or otherlocation sensor, can provide information identifying a location of theUAV. However, while the GPS receiver can provide a degree of accuracy,corrections to the location information can be required to ensure thatthe damaged area remains in focus. Thus, the UAV can optionally ensurethat the damaged area remains in the center of each captured image asthe UAV descends. To effect the above, the UAV can obtain one or moreimages illustrating the damaged area (e.g., obtained from the initialinspection described in FIG. 5), and can ensure that images being takenperiodically as it descends, comport with the obtained images (e.g., theimages obtained during the initial inspection). The UAV can execute oneor more visual classifiers to discriminate between portions of therooftop that correspond to the damaged area, and portions that lieoutside of the damaged are (e.g., the UAV can identify fixed features inthe images of the damaged area from the initial inspection, anddetermine corresponding locations in the periodically taken images). Asthe UAV descends, the UAV can correct the vertical descent (e.g.,provide corrections to the remaining axes).

Similarly, the UAV can optionally utilize a 3D model of the property androoftop to orient itself as it descends. For instance, the UAV canutilize images taken periodically as it descends, and using the locationinformation (e.g., GPS coordinates) with information describing a fieldof view of the camera, the UAV can determine its location within the 3Dmodel. For example, the UAV can determine, based off periodically takenimages, that the UAV is descending over a steeper portion of therooftop, or that the images show a portion of the rooftop, or property,that should not be visible. The UAV can then provide corrections to itsdescent, to ensure that the UAV remains vertically over the indicateddamaged area. Optionally, the UAV can utilize distance information(e.g., obtained using a LiDAR sensor), to provide fine corrections. Forinstance, as described above, the UAV can determine that based on itsdescent speed, the distance to the rooftop should be slightly different,and can provide a correction to move the UAV to a correct location overthe rooftop.

The UAV descends until reaching a threshold distance above the rooftop,and then ascends back to the safe altitude while optionally againactivating the sensors. The obtained real-world information can beprovided (e.g., over a wireless connection) to the user device of theoperator. The operator can indicate that one or more of the obtainedimages are of poor quality, or were never taken. Additionally,optionally, the UAV can determine whether a taken image was out offocus, or was never taken (e.g., the camera failed to fire). The UAV canre-take images of the damaged area (e.g., utilizing metadata of the poorquality image, the UAV can determine its height, travel back to thatheight, and re-take the image, or the UAV can access metadata indicatingthat it activated a camera at a certain height, but the camera failed toactivate).

The UAV travels to subsequent locations and performs inspections of theassociated damaged areas (block 606). As described above, with referenceto block 604, the UAV travels at the safe altitude to subsequentlocations, descends towards the rooftop and activates the includedsensors. Optionally, the UAV can utilize a 3D model of the property tonavigate over the rooftop to each subsequent damaged area (e.g., withoutascending to the safe altitude), for instance at the distance thresholdat which the UAV activates its sensors. Optionally, the UAV can utilizeone or more sensors to determine whether obstructions exist in itsflight path towards the subsequent damaged areas, and navigate aroundthe obstructions.

If the UAV determines, or the operator indicates, that one or more ofthe obtained images were of poor quality, or were failed to be taken,the UAV can travel to the associated height and re-take the image. Forinstance, if the UAV is descending down to a particular damaged area,and determines that images need to be re-taken at a different damagedarea, the UAV can complete the particular damaged area, ascend to thesafe altitude, and travel to the different damaged area. Optionally, theUAV can utilize a 3D model of the property to navigate over the rooftopback to the different damaged area (e.g., the 3D model can describeheight information of the rooftop, obstructions, and so on). Optionally,the UAV can utilize one or more sensors to determine whetherobstructions exist in its flight path towards the different damagedarea, and can navigate around them to travel to the different damagedarea.

The UAV travels to a safe landing location (block 608). After inspectingeach damaged area, the UAV can travel to the safe landing location(e.g., at the safe altitude, or at a different altitude if the UAV has a3D model of the property).

Report Generation and Presentation

After completing the subsequent inspection of the rooftop (e.g., asdescribed in FIG. 6), a report (e.g., an interactive report, or a staticreport) is generated that describes the inspection. The UAV, userdevice, or cloud system, can generate the report, and the report can beprovided to the operator, or a different reviewing user, forpresentation.

As described above, the report can include information describing theflight path of the UAV, such as a graphical representation of theproperty and flight path, with areas determined to be damaged identified(e.g., highlighted). The report can include textual informationdescribing the determined damage, sensor information (e.g., images) ofeach damaged area, and the presented report can enable the operator, orother reviewing user, to select damaged areas and access raw image andsensor data describing the damaged area. The operator can includetextual information describing each damaged area (e.g., the operator canindicate that a particular damaged area is damaged from hail) and visualinformation for each damaged area (e.g., the operator can circleportions of an image illustrating the damaged area to identifyparticular types of damage, and can assign each type of damage aparticular color). Additionally, as will be described below, optionallythe textual information and visual information can be automaticallydetermined (e.g., using one or more visual classifiers and computervision algorithms).

System Initiated Inspections

Optionally, the cloud system can determine that a property is to beinspected by determining that an inspection of the property was lastperformed greater than a threshold amount of time prior (e.g., 6 months,1 year). Additionally, the cloud system can determine that sensorinformation for a property (e.g., sensor information obtained in a priorjob) is of poor quality (e.g., the camera used by a UAV is substandardcompared to current cameras), or that it lacks particular sensor datafor the property (e.g., ultra-violet or infrared images), and candetermine that the property is to be inspected again. The cloud systemcan then store information describing the properties, and can schedulejobs associated with each property. The cloud system can prompt a userto review the properties, or trigger notifications to be immediatelypresented on a user device of a user (e.g., the user device can executean application in communication with the cloud system).

Automatic Damage Classification

As described above, the UAV, user device, or cloud system, can obtainreal-world information obtained during an initial inspection of arooftop, or during a subsequent inspection of the rooftop, and identifydamage (e.g., in substantially real-time as the UAV is flying, or afterthe UAV has landed). To effect the identification of damage, the UAV,user device, or cloud system, can utilize one or more visual classifierstrained on sensor information (e.g., images) that illustrate one or moretypes of damage (e.g., a particular classifier can be associated with aparticular type of damage). The below discussion describes hail damage,however other types of damage can be automatically identified, includingstorm damage, tornado damage, fire damage, and so on.

The discussion below will also describe the classifier being determinedby the cloud system (e.g., cloud system 120), however the user device,an outside system, the UAV, can also determine the classifier, updatethe classifier, and so on.

FIG. 7 is a flowchart of an example process 700 to determine a visualclassifier to identify rooftop damage. For convenience, the process 700will be described as being performed by a system of one or morecomputers (e.g., the cloud system 120).

The system obtains datasets that include sensor information describinghail damage of rooftops (block 702). The system can receive the datasetsfrom an outside entity (e.g., an insurance company) that determineswhether hail damage exists on rooftops of properties. Optionally, thesystem can obtain sensor information included in reports as describedabove, with damage identified by an operator. This report informationcan be obtained over a sufficient period of time to gather a largeenough dataset to train a classifier. The datasets can include sensorinformation describing undamaged areas of rooftops, and/or sensorinformation describing damaged areas of rooftops. The datasets canfurther include sensor information describing the same rooftops prior tobeing damaged (e.g., a before and after image of hail damage).

The datasets can be labeled according to specific damage illustrated,for instance, a reviewing user can identify that a particular imageillustrates hail damage, and the identification can be included asmetadata with the obtained datasets. Optionally, the datasets can beunlabeled, and can include sensor information of damaged, and undamaged,areas of rooftops. Additionally, the datasets can indicate a type ofroof material that was damaged, and the datasets can be organizedaccording to the type of roof material. For instance, hail damage on awood rooftop might be different than hail damage on a hot-tar-and-gravelroof.

The system trains visual classifier using the obtained dataset (block704). The system can use supervised, or unsupervised, trainingtechniques to train the visual classifier. As will be described, thetrained visual classifier will be used to detect hail damage.

As an example, the system can utilize a support vector machine (SVM) totrain the visual classifier. The system can obtain labeled datasets(e.g., the datasets can indicate whether the rooftop is damaged), andthe system can generate models describing features that characterize thesensor information as including damage. Additionally, the system cangenerate models using before and after sensor information of a samerooftop. For instance, the system can determine a difference between theimages (e.g., the system can process a before and after image to beoriented from a same perspective, correct for exposure, depth of field,lens effects, and can determine a difference in the images). Thedifference can be used when discriminating between features that informhail damage.

Furthermore, the system can utilize features that are known to beindicative of hail damage (e.g., a feature vector), and the visualclassifier can determine whether the features exist in received sensorinformation. For instance, the system can train the SVM to recognize thefeatures, and determine respective weightings of each feature to bestcharacterize sensor information as being indicative of a damaged area oran undamaged area.

As another example, the system can utilize unsupervised trainingtechniques, such as neural network modeling, k-means clustering, and soon. The system can use unsupervised training techniques to determinedistinct categories in the obtained datasets, and each category can belabeled by a user (e.g., a reviewing user), or the system can assignlikely levels of damage for each category. Additionally, theunsupervised training technique can utilize particular featuresindicative of hail damage to determine distinct categories in thedatasets. Thereafter, to determine levels of damage for each category,the system can assign higher levels of damage that correspond to greaterinclusions of the particular features.

A non-exhaustive list of features includes, cracks in ceramic/claytiles, broken corners of tiles, gaps between shingles, curled edges of arooftop, dents in a wooden rooftop, organisms growing on shingles, suchas moss or algae, shingle splitting (e.g., fresh split wood can belighter in color than older wood, indicative of recent hail damage),surface nail damage, spatter marks, whether shingles are lifted orturned, whether surface nails are raised above their normal level (e.g.,popping out), and so on. Additionally, the system can utilize featuresthat describe directionality of damage, for instance the system candetermine that where a rooftop has similar impact on similarlypositioned roof slopes (e.g., within a threshold distance from therooftop being inspected), there is likely hail damage on the rooftopbeing inspected. That is, if the system detects hail damage (e.g., usingthe above features) on a rooftop in a particular area along a particulardirection, the system can increase a likelihood of a different rooftopbeing damaged if it is within the particular area and along theparticular direction.

The system receives additional sensor information describing damagedareas (block 706). The system can obtain sensor information that hasbeen obtained during an inspection of a rooftop, and incorporate theobtained sensor information in improving the accuracy of the visualclassifier. For instance, an operator, or other reviewing user, canreceive information describing whether obtained images of an area of arooftop are indicative of damage (e.g., labeled by the visualclassifier). The operator, or other reviewing user, can indicate thatthe label is correct or incorrect. The system can utilize the updatedlabeled information to provide corrections to the visual classifier(e.g., as a feedback loop).

The system updates the visual classifier (block 708). As describedabove, the system can determine updates to the visual classifier. Theupdated visual classifier can be stored for use by UAVs, user devices,or by the cloud system.

FIG. 8 illustrates an example process for performing an inspection of arooftop using the visual classifier. For convenience, the process 800will be described as being performed by a system of one or moreprocessors (e.g., the UAV 100).

The UAV receives flight information describing a job to be performed(block 802). As described above, with reference to FIG. 5, the UAVreceives (e.g., from a user device, or from the cloud system)information describing a job.

The UAV ascends to safe altitude and performs inspection of the rooftopindicated in the job information (block 804). The UAV ascends to a safealtitude (e.g., indicated by an operator adjacent to the UAV) andperforms a flight pattern over the rooftop. As the UAV travels, itobtains sensor information of the rooftop. Optionally, the UAV canutilize a first included sensor (e.g., a camera with a wide-angle lens)to obtain sensor information, and then a second included sensor (e.g., acamera with a longer focal length) to obtain sensor information whendamage has been identified. Optionally, the UAV can use either the firsttype of sensor or the second type of sensor exclusively.

The UAV identifies a damaged area of the rooftop using the visualclassifier (block 806). The UAV can execute the visual classifier onreceived sensor information (e.g., images), and determine whether damageis evident. For instance, the visual classifier can determine whetherthe features described above are included in the obtained sensorinformation. The visual classifier can further utilize obtained sensorinformation of the rooftop at prior points in time (e.g., points in timewhen the rooftop did not have damage) to compare currently obtainedsensor information to the prior sensor information (e.g., as describedin FIG. 7). In this way, the specific features can be more clearlyestablished and illustrated for use by the visual classifier.Additionally, the UAV can utilize information identifying whether nearbyrooftops have been determined to have damage (e.g., rooftops at a sameslope, same height, and so on, as the presently being inspectedrooftop). The UAV can also utilize information describing a material therooftop is made of (e.g., wood, tile, clay) for use in identifyingdamage (e.g., some of the features indicative of hail damage arespecific to particular materials).

The UAV can then determine whether an area being inspected includesdamage. Since the damage might be localized to a particular area withinobtained sensor information (e.g., a small portion of the rooftop withineach obtained image), after determining the damaged area, the UAV cantravel directly above the damaged area, and maintain the damaged area infocus.

The UAV performs an inspection of the damaged area (block 808). Asdescribed above, the UAV can travel (e.g., fly) directly over thedamaged area, and then descend towards the damaged area. As the UAVdescends, it activates one or more sensors to obtain sensor informationdescribing the damaged area (e.g., as described above, with reference toFIG. 6).

The UAV performs an inspection of the remainder of the rooftop (block810). After completing the inspection of the damaged area, the UAVascends to the safe altitude, and determines whether damage is evidentelsewhere on the rooftop (e.g., using the visual classifier).Additionally, as described in FIG. 6, the UAV can utilize one or moresensors to detect obstructions, and can travel over the rooftop at lessthan the safe altitude using the sensors.

The UAV travels to a safe landing location (block 812). As described inFIG. 6, upon completion of the inspection, the UAV travels to the safelanding location. A report describing the inspection can be generated(e.g., by the UAV, user device, or cloud system, as described above).The report can include labels determined by the visual classifier fordamaged areas, and the visual classifier can be updated according towhether the visual classifier correctly characterized the rooftop.Additionally, the report can include particular features that wereevident in each damaged area (e.g., nail sticking above roof, algaedetected, and so on).

The processes described in FIGS. 5-6 and 8 can be modified to includefeatures from each other figure. For instance, the process of FIG. 8 canbe modified to include an initial inspection and a subsequentinspection, while still using a visual classifier.

Although in one embodiment of the invention, as described above, thesystem is primarily used to create and transmit job information (e.g.,as described in FIGS. 3-4) to a UAV or user device (e.g., ground controlstation), the UAV or user device can initiate the request for jobinformation from the system. That is, the UAV or user device (e.g., userof the user device) can arrive at a property location, and then requestjob information, or updated job information. For example, the UAV oruser device can determine its geospatial position via a GNSS receiver(using GPS, GLONASS, Galileo, or Beidou system). The UAV or user devicecan then transmit its location information to the system, along withother identifying information about the requesting device, such as itsUUID, or Mac address, etc. The system will then receive the request, anddetermine if updated or changed job information exists by comparing thedevice identifier to a database storing new or updated job information.If new or updated job information exists, then new or updated jobinformation will be transmitted from the system, and received by the UAVor user device. A confirmation acknowledging receipt of the jobinformation may then be transmitted from the UAV or user device to thesystem. The system will then update the database indicating that theparticular job information has been received. Moreover, the UAV or userdevice can supply the property location, and a new job request can besent to the system. The system may create new job information for theUAV or user device.

FIG. 9 illustrates a block diagram of an example Unmanned Aerial Vehicle(UAV) architecture for implementing the features and processes describedherein. A UAV primary processing system 900 can be a system of one ormore computers, or software executing on a system of one or morecomputers, which is in communication with, or maintains, one or moredatabases. The UAV primary processing system 900 can be a system of oneor more processors 935, graphics processors 936, I/O subsystem 934,logic circuits, analog circuits, associated volatile and/or non-volatilememory, associated input/output data ports, power ports, etc., and/orone or more software processing executing one or more processors orcomputers. The autopilot system 930 includes the IMU 932, processor 935,I/O subsystem 934, GPU 936, and various operating system 920, andmodules 920-929. Memory 918 may include non-volatile memory, such as oneor more magnetic disk storage devices, solid state hard drives, or flashmemory. Other volatile memory such a RAM, DRAM, SRAM may be used fortemporary storage of data while the UAV is operational. Databases maystore information describing UAV flight operations, flight plans,contingency events, geofence information, component information, andother information.

The UAV processing system may be coupled to one or more sensors, such asGNSS receivers 950 (e.g., a GPS, GLONASS, Galileo, or Beidou system),gyroscopes 956, accelerometers 958, pressure sensors (static ordifferential) 952, current sensors, voltage sensors, magnetometer,hydrometer, and motor sensors. The UAV may use an inertial measurementunit (IMU) 932 for use in navigation of the UAV. Sensors can be coupledto the processing system, or to controller boards coupled to the UAVprocessing system. One or more communication buses, such as a CAN bus,or signal lines, may couple the various sensor and components.

Various sensors, devices, firmware and other systems may beinterconnected to support multiple functions and operations of the UAV.For example, the UAV primary processing system 900 may use varioussensors to determine the vehicle's current geo-spatial location,attitude, altitude, velocity, direction, pitch, roll, yaw and/orairspeed and to pilot the vehicle along a specified route and/or to aspecified location and/or to control the vehicle's attitude, velocity,altitude, and/or airspeed (optionally even when not navigating thevehicle along a specific path or to a specific location).

The flight control module (also referred to as flight control engine)922 handles flight control operations of the UAV. The module interactswith one or more controllers 940 that control operation of motors 942and/or actuators 944. For example, the motors may be used for rotationof propellers, and the actuators may be used for flight surface controlsuch as ailerons, rudders, flaps, landing gear, and parachutedeployment.

The contingency module 924 monitors and handles contingency events. Forexample, the contingency module may detect that the UAV has crossed aborder of a geofence, and then instruct the flight control module toreturn to a predetermined landing location. Other contingency criteriamay be the detection of a low battery or fuel state, or malfunctioningof an onboard sensor, motor, or a deviation from the flight plan. Theforegoing is not meant to be limiting, as other contingency events maybe detected. In some instances, if equipped on the UAV, a parachute maybe deployed if the motors or actuators fail.

The mission module 929 processes the flight plan, waypoints, and otherassociated information with the flight plan. The mission module 929works in conjunction with the flight control module. For example, themission module may send information concerning the flight plan to theflight control module, for example lat/long waypoints, altitude, flightvelocity, so that the flight control module can autopilot the UAV.

The UAV may have various devices connected to it for data collection.For example, photographic camera 949, video cameras, infra-red camera,multispectral camera, and Lidar, radio transceiver, sonar, TCAS (trafficcollision avoidance system). Data collected by the devices may be storedon the device collecting the data, or the data may be stored onnon-volatile memory 918 of the UAV processing system 900.

The UAV processing system 900 may be coupled to various radios, andtransmitters 959 for manual control of the UAV, and for wireless orwired data transmission to and from the UAV primary processing system900, and optionally the UAV secondary processing system 902. The UAV mayuse one or more communications subsystems, such as a wirelesscommunication or wired subsystem, to facilitate communication to andfrom the UAV. Wireless communication subsystems may include radiotransceivers, and infrared, optical ultrasonic, electromagnetic devices.Wired communication systems may include ports such as Ethernet, USBports, serial ports, or other types of port to establish a wiredconnection to the UAV with other devices, such as a ground controlsystem, cloud-based system, or other devices, for example a mobilephone, tablet, personal computer, display monitor, other network-enableddevices. The UAV may use a light-weight tethered wire to a groundcontrol station for communication with the UAV. The tethered wire may beremoveably affixed to the UAV, for example via a magnetic coupler.

Flight data logs may be generated by reading various information fromthe UAV sensors and operating system and storing the information innon-volatile memory. The data logs may include a combination of variousdata, such as time, altitude, heading, ambient temperature, processortemperatures, pressure, battery level, fuel level, absolute or relativeposition, GPS coordinates, pitch, roll, yaw, ground speed, humiditylevel, velocity, acceleration, contingency information. This foregoingis not meant to be limiting, and other data may be captured and storedin the flight data logs. The flight data logs may be stored on aremovable media and the media installed onto the ground control system.Alternatively, the data logs may be wirelessly transmitted to the groundcontrol system or to the cloud system.

Modules, programs or instructions for performing flight operations,contingency maneuvers, and other functions may be performed with theoperating system. In some implementations, the operating system 920 canbe a real time operating system (RTOS), UNIX, LINUX, OS X, WINDOWS,ANDROID or other operating system. Additionally, other software modulesand applications may run on the operating system, such as a flightcontrol module 922, contingency module 924, application module 926, anddatabase module 928. Typically flight critical functions will beperformed using the UAV processing system 900. Operating system 920 mayinclude instructions for handling basic system services and forperforming hardware dependent tasks.

In addition to the UAV primary processing system 900, a secondaryprocessing system 902 may be used to run another operating system toperform other functions. A UAV secondary processing system 902 can be asystem of one or more computers, or software executing on a system ofone or more computers, which is in communication with, or maintains, oneor more databases. The UAV secondary processing system 902 can be asystem of one or more processors 994, graphics processors 992, I/Osubsystem 994 logic circuits, analog circuits, associated volatileand/or non-volatile memory, associated input/output data ports, powerports, etc., and/or one or more software processing executing one ormore processors or computers. Memory 970 may include non-volatilememory, such as one or more magnetic disk storage devices, solid statehard drives, flash memory. Other volatile memory such a RAM, DRAM, SRAMmay be used for storage of data while the UAV is operational.

Ideally modules, applications and other functions running on thesecondary processing system 902 will be non-critical functions innature, that is if the function fails, the UAV will still be able tosafely operate. In some implementations, the operating system 972 can bebased on real time operating system (RTOS), UNIX, LINUX, OS X, WINDOWS,ANDROID or other operating system. Additionally, other software modulesand applications may run on the operating system 972, such as anapplication module 974, database module 976. Operating system 902 mayinclude instructions for handling basic system services and forperforming hardware dependent tasks.

Also, controllers 946 may be used to interact and operate a payloaddevice 948, and other devices such as photographic camera 949, videocamera, infra-red camera, multispectral camera, stereo camera pair,Lidar, radio transceiver, sonar, laser ranger, altimeter, TCAS (trafficcollision avoidance system), ADS-B (Automatic dependentsurveillance-broadcast) transponder. Optionally, the secondaryprocessing system 902 may have coupled controllers to control payloaddevices.

For more details, see U.S. patent application Ser. No. 15/068,272,entitled “UNMANNED AERIAL VEHICLE ROOFTOP INSPECTION SYSTEM,” filed onMar. 11, 2016; U.S. patent application Ser. No. 15/068,327, entitled“UNMANNED AERIAL VEHICLE ROOFTOP INSPECTION SYSTEM,” filed on Mar. 11,2016; U.S. patent application Ser. No. 15/068,255, entitled “UNMANNEDAERIAL VEHICLE ROOFTOP INSPECTION SYSTEM,” filed on Mar. 11, 2016, eachof which is incorporated by reference herein in its entirety.

Each of the processes, methods, and algorithms described in thepreceding sections may be embodied in, and fully or partially automatedby, code modules executed by one or more computer systems or computerprocessors comprising computer hardware. The code modules (or “engines”)may be stored on any type of non-transitory computer-readable medium orcomputer storage device, such as hard drives, solid state memory,optical disc, and/or the like. The systems and modules may also betransmitted as generated data signals (for example, as part of a carrierwave or other analog or digital propagated signal) on a variety ofcomputer-readable transmission mediums, including wireless-based andwired/cable-based mediums, and may take a variety of forms (for example,as part of a single or multiplexed analog signal, or as multiplediscrete digital packets or frames). The processes and algorithms may beimplemented partially or wholly in application-specific circuitry. Theresults of the disclosed processes and process steps may be stored,persistently or otherwise, in any type of non-transitory computerstorage such as, for example, volatile or non-volatile storage.

In general, the terms “engine” and “module”, as used herein, refer tologic embodied in hardware or firmware, or to a collection of softwareinstructions, possibly having entry and exit points, written in aprogramming language, such as, for example, Java, Lua, C or C++. Asoftware module may be compiled and linked into an executable program,installed in a dynamic link library, or may be written in an interpretedprogramming language such as, for example, BASIC, Perl, or Python. Itwill be appreciated that software modules may be callable from othermodules or from themselves, and/or may be invoked in response todetected events or interrupts. Software modules configured for executionon computing devices may be provided on one or more computer readablemedia, such as a compact discs, digital video discs, flash drives, orany other tangible media. Such software code may be stored, partially orfully, on a memory device of the executing computing device. Softwareinstructions may be embedded in firmware, such as an EPROM. It will befurther appreciated that hardware modules may be comprised of connectedlogic units, such as gates and flip-flops, and/or may be comprised ofprogrammable units, such as programmable gate arrays or processors. Themodules described herein are preferably implemented as software modules,but may be represented in hardware or firmware. Generally, the modulesdescribed herein refer to logical modules that may be combined withother modules or divided into sub-modules despite their physicalorganization or storage. Electronic Data Sources can include databases,volatile/non-volatile memory, and any memory system or subsystem thatmaintains information.

The various features and processes described above may be usedindependently of one another, or may be combined in various ways. Allpossible combinations and subcombinations are intended to fall withinthe scope of this disclosure. In addition, certain method or processblocks may be omitted in some implementations. The methods and processesdescribed herein are also not limited to any particular sequence, andthe blocks or states relating thereto can be performed in othersequences that are appropriate. For example, described blocks or statesmay be performed in an order other than that specifically disclosed, ormultiple blocks or states may be combined in a single block or state.The example blocks or states may be performed in serial, in parallel, orin some other manner. Blocks or states may be added to or removed fromthe disclosed example embodiments. The example systems and componentsdescribed herein may be configured differently than described. Forexample, elements may be added to, removed from, or rearranged comparedto the disclosed example embodiments.

Conditional language used herein, such as, among others, “can,” “could,”“might,” “may,” “for example,” and the like, unless specifically statedotherwise, or otherwise understood within the context as used, isgenerally intended to convey that certain embodiments include, whileother embodiments do not include, certain features, elements and/orsteps. Thus, such conditional language is not generally intended toimply that features, elements and/or steps are in any way required forone or more embodiments or that one or more embodiments necessarilyinclude logic for deciding, with or without author input or prompting,whether these features, elements and/or steps are included or are to beperformed in any particular embodiment. The terms “comprising,”“including,” “having,” and the like are synonymous and are usedinclusively, in an open-ended fashion, and do not exclude additionalelements, features, acts, operations, and so forth. Also, the term “or”is used in its inclusive sense (and not in its exclusive sense) so thatwhen used, for example, to connect a list of elements, the term “or”means one, some, or all of the elements in the list. Conjunctivelanguage such as the phrase “at least one of X, Y and Z,” unlessspecifically stated otherwise, is otherwise understood with the contextas used in general to convey that an item, term, etc. may be either X, Yor Z. Thus, such conjunctive language is not generally intended to implythat certain embodiments require at least one of X, at least one of Yand at least one of Z to each be present.

The term “a” as used herein should be given an inclusive rather thanexclusive interpretation. For example, unless specifically noted, theterm “a” should not be understood to mean “exactly one” or “one and onlyone”; instead, the term “a” means “one or more” or “at least one,”whether used in the claims or elsewhere in the specification andregardless of uses of quantifiers such as “at least one,” “one or more,”or “a plurality” elsewhere in the claims or specification.

The term “comprising” as used herein should be given an inclusive ratherthan exclusive interpretation. For example, a general purpose computercomprising one or more processors should not be interpreted as excludingother computer components, and may possibly include such components asmemory, input/output devices, and/or network interfaces, among others.

While certain example embodiments have been described, these embodimentshave been presented by way of example only, and are not intended tolimit the scope of the disclosure. Thus, nothing in the foregoingdescription is intended to imply that any particular element, feature,characteristic, step, module, or block is necessary or indispensable.Indeed, the novel methods and systems described herein may be embodiedin a variety of other forms; furthermore, various omissions,substitutions, and changes in the form of the methods and systemsdescribed herein may be made without departing from the spirit of theinventions disclosed herein. The accompanying claims and theirequivalents are intended to cover such forms or modifications as wouldfall within the scope and spirit of certain of the inventions disclosedherein.

Any process descriptions, elements, or blocks in the flow diagramsdescribed herein and/or depicted in the attached figures should beunderstood as potentially representing modules, segments, or portions ofcode which include one or more executable instructions for implementingspecific logical functions or steps in the process. Alternateimplementations are included within the scope of the embodimentsdescribed herein in which elements or functions may be deleted, executedout of order from that shown or discussed, including substantiallyconcurrently or in reverse order, depending on the functionalityinvolved, as would be understood by those skilled in the art.

It should be emphasized that many variations and modifications may bemade to the above-described embodiments, the elements of which are to beunderstood as being among other acceptable examples. All suchmodifications and variations are intended to be included herein withinthe scope of this disclosure. The foregoing description details certainembodiments of the invention. It will be appreciated, however, that nomatter how detailed the foregoing appears in text, the invention can bepracticed in many ways. As is also stated above, it should be noted thatthe use of particular terminology when describing certain features oraspects of the invention should not be taken to imply that theterminology is being re-defined herein to be restricted to including anyspecific characteristics of the features or aspects of the inventionwith which that terminology is associated.

What is claimed is:
 1. A computerized method performed by a systemcomprising one or more processors, the method comprising: displaying,via an interface, a first graphical representation of a structure to beinspected, the first graphical representation comprising one or moreimages describing an aerial view of the structure; determining a flightpattern according to which an unmanned aerial vehicle (UAV) is tonavigate above the structure and obtain sensor data describing an aerialview of the structure, the flight pattern including waypoints;instructing the UAV to perform the flight pattern and navigate the UAVaccording to the flight pattern, and obtain sensor data; receivingsensor data obtained by the UAV during performance of the flightpattern, wherein the sensor data includes multiple digital imagesdescribing an aerial view depicting a rooftop of the structure, andwherein a subset of the digital images include greater than a thresholdlevel of detail of particular areas of the rooftop; generating a secondgraphical representation of the structure based on the multiple digitalimages, wherein the second graphical representation comprises compositeimagery of the rooftop based on the multiple digital images; determiningthe occurrence of damaged areas of the rooftop in the multiple digitalimages; and displaying, via a user interface, the second graphicalrepresentation of the rooftop with damaged areas of the rooftopidentified in the second graphical representation, wherein the damagedareas of the rooftop are represented by a graphical indication of therespective damaged areas, wherein the user interface is configured toreceive selection of a damaged area identified in the second graphicalrepresentation and display one or more digital images that includegreater than the threshold level of detail of the damaged area.
 2. Themethod of claim 1, wherein determining the occurrence of damaged areasof the rooftop comprises: executing a visual classifier trained toidentify rooftop damage, that a particular area of the rooftopidentified from the obtained sensor information is indicated as beingdamaged.
 3. The method of claim 1, wherein determining whether surfacedamage has occurred about a particular area of the rooftop comprises:determining whether a feature indicative of rooftop damage exists in adigital image; and weighting the feature to determine that the featureis indicative of a damaged area or an undamaged area.
 4. The method ofclaim 1, further comprising training a visual classifier based on afirst dataset of images comprising examples of rooftop damage.
 5. Themethod of claim 1, further comprising: determining based on the aerialimages, a material composition of the rooftop, wherein the materialcomposition for the rooftop is one of wood, tile or clay.
 6. The methodof claim 1, wherein the type of rooftop damage identified is one of haildamage, cracked tiles, broken tiles, surface dents or organism growth.7. The method of claim 1, further comprising: receiving, via a userinterface, positions about the second graphical representation of therooftop indicating locations of damaged areas.
 8. The method of claim 1,wherein the positions above the structure are set at an altitude suchthat a required image pixel resolution is obtained.
 9. The method ofclaim 1, further comprising: displaying, via the interface, a graphicalrepresentation of the flight pattern above the first graphicalrepresentation of the property.
 10. The method of claim 1, furthercomprising: determining the type of damage; and assigning a descriptionto the type of damage.
 11. A system comprising one or more processors,and a computer storage media storing instructions, that when executed bythe one or more processors, cause the one or more processors to performoperations comprising: displaying, via an interface, a first graphicalrepresentation of a structure to be inspected, the first graphicalrepresentation comprising one or more images describing an aerial viewof the structure; determining a flight pattern according to which anunmanned aerial vehicle (UAV) is to navigate above the structure andobtain sensor data describing an aerial view of the structure, theflight pattern including waypoints; instructing the UAV to perform theflight pattern and navigate the UAV according to the flight pattern, andobtain sensor data; receiving sensor data obtained by the UAV duringperformance of the flight pattern, wherein the sensor data includesmultiple digital images describing an aerial view depicting a rooftop ofthe structure, and wherein a subset of the digital images are detailedimages; generating a second graphical representation of the structurebased on the multiple digital images, wherein the second graphicalrepresentation comprises composite imagery of the rooftop based on themultiple digital images; determining the occurrence of damaged areas ofthe rooftop in the multiple digital images; and displaying, via a userinterface, the second graphical representation of the rooftop withdamaged areas of the rooftop identified in the second graphicalrepresentation, wherein the damaged areas of the rooftop are representedby a graphical indication of the respective damaged areas, wherein theuser interface is configured to receive selection of a damaged area anddisplay one or more detailed images of the damaged area.
 12. The systemof claim 11, wherein determining the occurrence of damaged areas of therooftop comprises: executing a visual classifier trained to identifyrooftop damage, that a particular area of the rooftop is indicated asbeing damaged.
 13. The system of claim 12, further comprising theoperations of: training the visual classifier based on a first datasetof images comprising examples of rooftop damage.
 14. The system of claim11, wherein determining the occurrence of damaged areas of the rooftopcomprises: determining whether a feature indicative of rooftop damageexists in a digital image; and weighting the feature to determine thatthe feature is indicative of a damaged area or an undamaged area. 15.The system of claim 11, further comprising the operations of:determining based on the aerial images, a material composition of therooftop, wherein the material composition for the rooftop is one ofwood, tile or clay.
 16. The system of claim 11, wherein the type ofrooftop damage identified is one of hail damage, cracked tiles, brokentiles, surface dents or organism growth.
 17. The system of claim 11,further comprising the operations of: receiving, via a user interface,positions about the second graphical representation of the rooftopindicating locations of damaged areas of the rooftop.
 18. The system ofclaim 11, wherein waypoints above the structure are set at an altitudesuch that a required image pixel resolution for the multiple digitalimages of the rooftop is obtained.
 19. The system of claim 11, furthercomprising the operations of: displaying, via the interface, a graphicalrepresentation of the flight pattern above the first graphicalrepresentation of the property.
 20. The system of claim 11, furthercomprising the operations of: determining the type of damage; andassigning a description to the type of damage.
 21. A computer storagemedium comprising instructions that when executed by one or moreprocessors, cause the processors to perform operations comprising:displaying, via an interface, a first graphical representation of astructure to be inspected, the first graphical representation comprisingone or more images describing an aerial view of the structure;determining a flight pattern according to which an unmanned aerialvehicle (UAV) is to navigate above the structure and obtain sensor datadescribing an aerial view of the structure, the flight pattern includingwaypoints; instructing the UAV to perform the flight pattern andnavigate the UAV according to the flight pattern, and obtain sensordata; receiving sensor data obtained by the UAV during performance ofthe flight pattern, wherein the sensor data includes multiple digitalimages describing an aerial view depicting a rooftop of the structure,and wherein a subset of the digital images include greater than athreshold level of detail of particular areas of the rooftop; generatinga second graphical representation of the structure based on the multipledigital images, wherein the second graphical representation comprisescomposite imagery of the rooftop based on the multiple digital images;determining the occurrence of damaged areas of the rooftop in themultiple digital images; and displaying, via a user interface, thesecond graphical representation of the rooftop with damaged areas of therooftop identified in the second graphical representation, wherein thedamaged areas of the rooftop are represented by a graphical indicationof the respective damaged areas, wherein the user interface isconfigured to receive selection of a damaged area and display one ormore digital images that include greater than the threshold level ofdetail of the damaged area.
 22. The computer storage medium of claim 21,wherein determining the occurrence of damaged areas of the rooftopcomprises: executing a visual classifier trained to identify rooftopdamage, that a particular area of the rooftop is indicated as beingdamaged.
 23. The computer storage medium of claim 22, further comprisingthe operations of: training a visual classifier based on a first datasetof images comprising examples of rooftop damage.
 24. The computerstorage medium of claim 21, wherein determining the occurrence ofdamaged areas of the rooftop comprises: determining whether a featureindicative of rooftop damage exists in a digital image; and weightingthe feature to determine that the feature is indicative of a damagedarea or an undamaged area.
 25. The computer storage medium of claim 21,further comprising the operations of: determining based on the aerialimages, a material composition of the rooftop, wherein the materialcomposition for the rooftop is one of wood, tile or clay.
 26. Thecomputer storage medium of claim 21, wherein the type of rooftop damageidentified is one of hail damage, cracked tiles, broken tiles, surfacedents or organism growth.
 27. The computer storage medium of claim 21,further comprising the operations of: receiving, via a user interface,positions about the second graphical representation of the rooftopindicating locations of damaged areas of the rooftop.
 28. The computerstorage medium of claim 21, wherein waypoints above the structure areset at an altitude such that a required image pixel resolution for themultiple digital images of the rooftop is obtained.
 29. The computerstorage medium of claim 21, further comprising the operations of:displaying, via the interface, a graphical representation of the flightpattern above the first graphical representation of the property. 30.The computer storage medium of claim 21, further comprising theoperations of: determining the type of damage; and assigning adescription to the type of damage.